Imaging device and imaging system

ABSTRACT

An imaging apparatus according to one embodiment of the present invention includes: a lens optical system having first to third regions, the first region transmitting light of a first wavelength band, the second region transmitting light of the first wavelength band and having optical characteristics for providing a different focusing characteristic from a focusing characteristic associated with rays transmitted through the first region, and the third region transmitting light of a second wavelength band; an imaging device on which light having passed through the lens optical system is incident, the imaging device having a plurality of first to third pixels; and a microlens array causing light having passed through the first region to enter the plurality of first pixels, light having passed through the second region to enter the plurality of second pixels, and light having passed through the third region to enter the plurality of third pixels.

This is a continuation of International Application No.PCT/JP2012/007668, with an international filing date of Nov. 29, 2012,which claims priority of Japanese Patent Applications No. 2011-261594filed on Nov. 30, 2011 and No. 2011-274680 filed on Dec. 15, 2011, theentire contents of which are hereby incorporated by reference.

BACKGROUND

1. Technical Field

The present disclosure relates to an imaging apparatus such as a camera.

2. Description of the Related Art

In recent years, distance measuring apparatuses which measure a distancefrom a subject (target of distance measurement) based on a parallaxbetween a plurality of imaging optical systems are used for themeasurement of vehicular gaps between automobiles and in cameraauto-focus systems and three-dimensional shape measurement systems.

In such distance measuring apparatuses, a pair of imaging opticalsystems that are positioned in right-left or upper-lower relationshipcreate images in their respective imaging regions, such that a distancefrom the subject is detected from the parallax between these imagesbased on triangulation.

As a method of measuring a distance from a subject with a single imagingoptical system, a DFD (Depth From Defocus) technique is known. Althoughthe DFD technique is a technique for calculating a distance by analyzingan amount of blur of an acquired image, it is impossible to know from asingle image whether something is a pattern possessed by the subjectitself, or a blur caused by subject distance; therefore, a technique ofestimating a distance from a plurality of images is adopted (PatentDocument 1 (Japanese Patent No. 3110095), Non-Patent Document 1 (Xue Tu,Youn-sik Kang and Murali Subbarao Two- and Three-Dimensional Methods forInspection and Metrology V. Edited by Huang, Peisen S. Proceedings ofthe SPIE, Volume 6762, pp. 676203 (2007))).

SUMMARY

However, in the aforementioned conventional techniques, downsizing andcost reduction of the imaging apparatus, improvement in the accuracy ofdistance measurement, and so on have been desired. One non-limiting andexemplary embodiment of the present disclosure provides an imagingapparatus which, in small size, is capable of accurate distancemeasurement.

An imaging apparatus according to one embodiment of the presentinvention comprises: a lens optical system having a first region, asecond region, and a third region, the first region transmitting lightof a first wavelength band, the second region transmitting light of thefirst wavelength band and having optical characteristics for providing adifferent focusing characteristic from a focusing characteristicassociated with rays transmitted through the first region, and the thirdregion transmitting light of a second wavelength band different from thefirst wavelength band; an imaging device on which light having passedthrough the lens optical system is incident, the imaging device having aplurality of first, second, and third pixels; and a microlens arraydisposed between the lens optical system and the imaging device, themicrolens array causing light having passed through the first region toenter the plurality of first pixels, light having passed through thesecond region to enter the plurality of second pixels, and light havingpassed through the third region to enter the plurality of third pixels.

With a distance measuring apparatus according to one embodiment of thepresent invention, it is possible to achieve accurate distancemeasurement by using a single imaging system.

These general and specific aspects may be implemented using a system, amethod, and a computer program, and any combination of systems, methods,and computer programs.

Additional benefits and advantages of the disclosed embodiments will beapparent from the specification and Figures. The benefits and/oradvantages may be individually provided by the various embodiments andfeatures of the specification and drawings disclosure, and need not allbe provided in order to obtain one or more of the same.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram showing Embodiment 1 of an imagingapparatus A according to the present invention.

FIG. 2 is a front view showing an optical element L1 according toEmbodiment 1 of the present invention as viewed from the subject side.

FIG. 3 is a perspective view of an array optical device K according toEmbodiment 1 of the present invention.

FIG. 4A is an enlarged diagram showing the array optical device K andimaging device N shown in FIG. 1.

FIG. 4B is a diagram showing relative positioning between the arrayoptical device K and pixels on the imaging device N.

FIG. 5 is a graph showing a spherical aberration associated with rayspassing through an optical region D1 and an optical region D2 inEmbodiment 1 of the present invention.

FIG. 6 is a conceptual diagram of point spread distributions fordifferent subject distances.

FIG. 7 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 1 of thepresent invention.

FIG. 8( a) to FIG. 8( c) each show a luminance distribution in an imageblock sized 16×16; and FIG. 8( d) to FIG. 8( f) show frequency spectrumsobtained by applying a two-dimensional Fourier transform to therespective image blocks shown in FIG. 8( a) to FIG. 8( c).

FIG. 9A is a diagram showing a subject, which is a chart of white andblack.

FIG. 9B is a diagram showing a cross section in the luminance of thesubject of FIG. 9A.

FIG. 9C is a diagram showing a cross section in the luminance of animage which is captured by the imaging apparatus A of FIG. 1 for eachcolor;

FIG. 9D is a diagram showing a second-order differentiation of the G1(green) and R(red) luminance in FIG. 9C.

FIG. 9E is a diagram showing a cross section in the luminance when thesecond-order differentiation of FIG. 9D is subtracted from the G2(green) and B(blue) luminance in FIG. 9C.

FIG. 10 is a graph showing another relationship between subject distanceand sharpness (how sharp an image is) according to Embodiment 1 of thepresent invention.

FIG. 11 is a graph showing another relationship between subject distanceand sharpness (how sharp an image is) according to Embodiment 1 of thepresent invention.

FIG. 12 is a cross-sectional view showing the imaging apparatus A ofEmbodiment 1.

FIG. 13 is a diagram showing another imaging apparatus according toEmbodiment 1.

FIG. 14 is a front view showing another optical element L1 according toEmbodiment 1 of the present invention as viewed from the subject side.

FIG. 15 is a front view showing another optical element L1 according toEmbodiment 1 of the present invention as viewed from the subject side.

FIG. 16A and FIG. 16B are enlarged diagrams each showing an arrayoptical device K and an imaging device N according to Embodiment 2 ofthe present invention.

FIG. 17A and FIG. 17B are front views showing relative positioningbetween different optical regions and a light-shielding member accordingto Embodiment 3 of the present invention.

FIG. 18 is a schematic diagram showing Embodiment 4 of an imagingapparatus A according to the present invention.

FIG. 19 is a front view showing an optical element L1 according toEmbodiment 4 of the present invention as viewed from the subject side.

FIG. 20 is a perspective view of an array optical device K according toEmbodiment 4 of the present invention.

FIG. 21A is an enlarged diagram showing the array optical device K andimaging device N shown in FIG. 18.

FIG. 21B is a diagram showing relative positioning between the arrayoptical device K and pixels on the imaging device N.

FIG. 22 is a flowchart for signal processing sections according toEmbodiment 4 of the present invention.

FIG. 23 is a graph showing a spherical aberration associated with rayspassing through an optical region D1 and an optical region D2 inEmbodiment 4 of the present invention.

FIG. 24 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 4 of thepresent invention.

FIG. 25( a) to FIG. 25( c) each show a luminance distribution in animage block sized 16×16; and FIG. 25( d) to FIG. 25( f) show frequencyspectrums obtained by applying a two-dimensional Fourier transform tothe respective image blocks shown in FIG. 25( a) to FIG. 25( c).

FIG. 26A is a diagram showing a subject, which is a chart of white andblack.

FIG. 26B is a diagram showing a cross section in the luminance of thesubject of FIG. 26A.

FIG. 26C is a diagram showing a cross section in the luminance of animage which is captured by the imaging apparatus A of FIG. 1.

FIG. 26D is a diagram showing second-order differentiation of the G1luminance in FIG. 26C.

FIG. 26E is a diagram showing a cross section in the luminance when thesecond-order differentiation of FIG. 26D is subtracted from the G2luminance in FIG. 26C.

FIG. 27 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 4 of thepresent invention.

FIG. 28 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 4 of thepresent invention.

FIG. 29( a) is a diagram showing a subject image according to Embodiment4 of the present invention; and FIG. 29( b) is a diagram showing a depthmap of the subject image of FIG. 29( a).

FIG. 30 is a distribution diagram of a PSF (point spread function)cross-sectional intensity, expressed as a Gaussian distribution,according to Embodiment 4 of the present invention.

FIG. 31A and FIG. 31B are diagrams showing a relationship betweensubject positions and PSFs according to Embodiment 4 of the presentinvention.

FIG. 32( a) to FIG. 32( c) are diagrams showing PSF two-dimensional dataaccording to Embodiment 4 of the present invention.

FIG. 33A and FIG. 33B are distribution diagrams of PSF two-dimensionalintensity according to Embodiment 4 of the present invention.

FIG. 34 is a diagram showing a refocused image of the subject image ofFIG. 29( a) based on the depth map of FIG. 29( b), according toEmbodiment 4 of the present invention.

FIG. 35 is a diagram showing a relationship between subject positionsand PSFs according to Embodiment 5 of the present invention.

FIG. 36 is a schematic diagram showing Embodiment 6 of an imagingapparatus A according to the present invention.

FIG. 37A is an enlarged diagram showing the array optical device K andimaging device N shown in FIG. 36.

FIG. 37B is a diagram showing relative positioning between the arrayoptical device K and pixels on the imaging device N.

FIG. 38 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 6 of thepresent invention.

FIG. 39A is a diagram showing a subject, which is a chart of white andblack.

FIG. 39B is a diagram showing a cross section in the luminance of thesubject of FIG. 39A.

FIG. 39C is a diagram showing a cross section in the luminance of animage which is captured by the imaging apparatus A of FIG. 36 for eachcolor.

FIG. 39D is a diagram showing a second-order differentiation of the G1(green) and R(red) luminance in FIG. 39C.

FIG. 39E is a diagram showing a cross section in the luminance of whenthe second-order differentiation of FIG. 39D is subtracted from the G2(green) and B(blue) luminance in FIG. 39C.

FIG. 40 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 6 of thepresent invention.

FIG. 41 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 6 of thepresent invention.

FIG. 42 is a front view showing an optical element L1 according toEmbodiment 7 of the present invention as viewed from the subject side.

FIG. 43 is a perspective view of an array optical device K according toEmbodiment 7 of the present invention.

FIG. 44A is an enlarged diagram showing of the array optical device Kand imaging device N according to Embodiment 7 of the present invention.

FIG. 44B is a diagram showing relative positioning between the arrayoptical device K and pixels on the imaging device N.

FIG. 45 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 7 of thepresent invention.

FIG. 46 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 7 of thepresent invention.

FIG. 47 is a graph showing a relationship between subject distance andsharpness (how sharp an image is) according to Embodiment 7 of thepresent invention.

FIG. 48 is a schematic diagram showing Embodiment 8 of an imagingapparatus A according to the present invention.

FIG. 49A is an enlarged diagram showing the neighborhood of an imagingplane when crosstalk occurs in an embodiment of the present invention.

FIG. 49B is an enlarged diagram showing the neighborhood of the imagingplane when crosstalk is reduced.

FIG. 50( a 1) is a perspective view showing a microlens array having arotation-asymmetric shape with respect to the optical axis; FIG. 50( a2) is a diagram showing contours of the microlens array shown in FIG.50( a 1); FIG. 50( a 3) is a diagram showing a ray tracing simulationresult when the microlens shown in FIG. 50( a 1) and FIG. 50( a 2) isapplied to the array optical device according to the present invention;FIG. 50( b 1) is a perspective view showing a microlens array having arotation symmetric shape with respect to the optical axis; FIG. 50( b 2)is a diagram showing contours of the microlens array shown in FIG. 50( b1); and FIG. 50( b 3) is a diagram showing a ray tracing simulationresult when the microlens shown in FIG. 50( b 1) and FIG. 50( b 2) isapplied to the array optical device according to an embodiment of thepresent invention.

FIG. 51 is a diagram showing another embodiment of a filter array on theimaging device according to an embodiment of the present invention.

DETAILED DESCRIPTION

According to studies under the present inventors, in conventionalconstructions using a plurality of imaging optical systems, the imagingapparatus will increase in size and cost. Moreover, the need to ensurematching characteristics between the plurality of imaging opticalsystems and a highly precise parallelism between the optical axes of thetwo imaging optical systems makes fabrication difficult. Furthermore,the need for a calibration step for determining camera parameters willpresumably require a large number of steps.

In the DFD technique, as disclosed in Patent Document 1 and Non-PatentDocument 1, a distance from the subject can be calculated with a singleimaging optical system. However, in the methods of Patent Document 1 andNon-Patent Document 1, it is necessary to obtain a plurality of imagesvia time division while varying the distance from the subject at whichfocus is achieved (focusing distance). When such a technique is appliedto a motion video, discrepancies between images will occur due to timelags in imaging, thus resulting in a problem of lowered accuracy ofdistance measurement.

Moreover, Patent Document 1 discloses an imaging apparatus which splitsthe optical path with a prism so that imaging occurs on two imagingplanes with different back focuses, thereby making it possible tomeasure the distance from a subject through a single instance ofimaging. However, such a method requires two imaging planes, thusleading to a problem in that the imaging apparatus is increased in sizeand the cost is considerably increased.

In view of the above problems, the inventors have arrived at a novelimaging apparatus. In outline, embodiments of the present invention maybe as follows.

An imaging apparatus according to one embodiment of the presentinvention comprises: a lens optical system having a first region, asecond region, and a third region, the first region transmitting lightof a first wavelength band, the second region transmitting light of thefirst wavelength band and having optical characteristics for providing adifferent focusing characteristic from a focusing characteristicassociated with rays transmitted through the first region, and the thirdregion transmitting light of a second wavelength band different from thefirst wavelength band; an imaging device on which light having passedthrough the lens optical system is incident, the imaging device having aplurality of first, second, and third pixels; and a microlens arraydisposed between the lens optical system and the imaging device, themicrolens array causing light having passed through the first region toenter the plurality of first pixels, light having passed through thesecond region to enter the plurality of second pixels, and light havingpassed through the third region to enter the plurality of third pixels.

The lens optical system may further have a fourth region transmittinglight of a third wavelength band different from the first and secondwavelength bands; the imaging device may further include a plurality offourth pixels; and the microlens array may cause light having passedthrough the fourth region to enter the plurality of fourth pixels.

The first, second, and third regions may be regions divided around anoptical axis of the lens optical system.

In the lens optical system, a focusing characteristic associated withrays transmitted through the third region and the fourth region may beidentical to either a focusing characteristic associated with raystransmitted through the first region or a focusing characteristicassociated with rays transmitted through a second region.

Rays may be incident on the first, second, and third regions through asingle instance of imaging.

The first and second regions may allow rays of the green band to passthrough, the third region may allow rays of the blue band to passthrough, and the fourth region may allow rays of the red band to passthrough.

When a subject distance is within a predetermined range, a point spreaddistribution created by light entering the first region may besubstantially constant, and a point spread distribution created by lightentering the second region may vary in accordance with distance from asubject.

A surface of the first region and a surface of the second region mayhave mutually different radii of curvature.

The plurality of first and second pixels may respectively generate firstand second luminance information through a single instance of imaging;and the imaging apparatus may further comprise a first signal processingsection for generating a first image and a second image by using thefirst and second luminance information.

The first signal processing section may include a sharpness detectionsection for detecting a sharpness of at least one pixel component,within luminance information of the plurality of first to fourth pixels,for each predetermined region in an image; and based on a component of ahighest sharpness among the respective sharpnesses, a luminanceinformation component of another pixel may be sharpened.

By using a previously stored point spread function, the first signalprocessing section may perform a restoration process for an image whichis formed based on luminance information of a pixel reached by lightentering the first region, and generate a restored sharpened image.

The first signal processing section may use a single said point spreadfunction to perform a restoration process for all regions of an imagewhich is formed based on luminance information of a pixel reached bylight entering the first region, and generate a restored sharpenedimage.

The first signal processing section may include a sharpness detectionsection for detecting a sharpness for each predetermined region in therestored sharpened image, and, based on a sharpness of eachpredetermined region in the restored sharpened image, sharpen aluminance information component of another pixel.

The imaging apparatus may further comprise a second signal processingsection for calculating a distance from a subject, wherein the secondsignal processing section may calculate a distance from the subject byusing the first image and the second image.

When the subject distance is within a certain range, a value of a ratiobetween a sharpness of the first image and a sharpness of the secondimage may have a correlation with the distance from the subject; and thesecond signal processing section may calculate the distance from thesubject based on the correlation and the ratio between the sharpness ofthe first image and the sharpness of the second image.

The first signal processing section may include a contrast detectionsection for detecting a contrast of the first image obtained from theplurality of first pixels and a contrast of the second image obtainedfrom the plurality of second pixels and; when the subject distance iswithin a certain range, a ratio between the contrast of the first imageand the contrast of the second image may have a correlation with thesubject distance; and the second signal processing section may calculatethe distance from the subject based on the correlation, the contrast ofthe first image and the contrast of the second image.

The second signal processing section may calculate the distance from thesubject by using luminance information of an image obtained throughaddition of the first image and the second image and luminanceinformation of the first image or the second image.

When the subject distance is within a certain range, a point spreadfunction derived from an image which is formed from the restoredsharpened image and light entering the second region may have acorrelation with the subject distance; and the second signal processingsection may calculate the distance from the subject based on thecorrelation and the point spread function.

The second region, the third region, and the fourth region may havemutually different optical powers; and focusing positions of lighttransmitted through the second region, the third region, and the fourthregion may be closer to one another than when the second region, thethird region, and the fourth region have an equal optical power to oneanother.

The imaging apparatus may further comprise a light-shielding memberprovided at a boundary between the first region and the second region.

The lens optical system may further include a stop; and the first regionand the second region may be disposed near the stop.

The second signal processing section may calculate a subject distancefor each predetermined region in an image; and the imaging apparatus mayfurther comprise a third signal processing section for generating arefocused image by using the subject distance for each predeterminedregion calculated by the second signal processing section.

The second signal processing section may generate a point spreadfunction for each subject distance by using a subject distance for eachpredetermined region.

Along the subject distance direction, an intensity change in the pointspread function may decrease away from at least one best focus position,the at least one best focus position defining a subject distance atwhich an intensity change in the point spread function takes a localmaximum.

The at least one best focus position may be an externally input positionor a position determined by the second signal processing section.

The third signal processing section may generate the refocused image byusing the subject distance for each predetermined region and the pointspread function.

The point spread function may be a Gaussian function.

The third signal processing section may generate the refocused image byperforming a convolution calculation for the point spread function usinga Fourier transform for each predetermined region.

The third signal processing section may generate the refocused image byperforming a spatial filter process based on the subject distance foreach predetermined region.

The at least one best focus position may exist in plurality anddiscretely.

The imaging apparatus may further comprise first to fourth filters nearthe lens optical system, the first to fourth filters being providedrespectively in the first region, the second region, the third region,and the fourth region, wherein, the first filter may transmit light ofthe first wavelength band; the second filter may transmit light of thefirst wavelength band; the third filter may transmit light of the secondwavelength band; and the fourth filter may transmit light of the thirdwavelength band.

The lens optical system may further comprise a stop; and the first tofourth filters may be disposed near the stop.

An imaging system according to one embodiment of the present inventioncomprises: the above imaging apparatus; and a first signal processingapparatus for generating a color image, wherein the first signalprocessing apparatus generates the color image by using luminanceinformation of the plurality of first pixels, the plurality of secondpixels, the plurality of third pixels, and the plurality of fourthpixels obtained through a single instance of imaging.

The imaging system may further comprise a second signal processingapparatus for calculating a distance from a subject, wherein the secondsignal processing apparatus may calculate a distance from the subject byusing the luminance information of the plurality of first pixels and theplurality of second pixels obtained through the single instance ofimaging.

An imaging system according to another embodiment of the presentinvention comprises an imaging apparatus and a signal processingapparatus, wherein the imaging apparatus includes: a lens optical systemhaving a first region and a second region, the second region havingoptical characteristics for providing a different focusingcharacteristic from a focusing characteristic associated with rayshaving passed through the first region; an imaging device on which lighthaving passed through the lens optical system is incident, the imagingdevice at least having a plurality of first pixels and a plurality ofsecond pixels; and an array optical device disposed between the lensoptical system and the imaging device, the array optical device causinglight having passed through the first region to enter the plurality offirst pixels and light having passed through the second region to enterthe plurality of second pixels, and the signal processing apparatusincludes: a first signal processing section for calculating a subjectdistance for each predetermined region in a captured image, by usingluminance information of a first image obtained from the plurality offirst pixels and a second image obtained from the plurality of secondpixels; and a second signal processing section for generating arefocused image by using the subject distance for each predeterminedregion calculated by the first signal processing section.

With an imaging apparatus and imaging system according to the aboveembodiment, by using a single optical system, it is possible to acquireluminance information for color image output and subject distancemeasurement through a single instance of imaging. This is unlike in animaging apparatus having a plurality of imaging optical systems, whereit would be necessary to ensure matching characteristics and positionsbetween the plurality of imaging optical systems. Moreover, even if thesubject position changes with lapse of time during the shooting of amotion video, an accurate distance from the subject can be measured.Moreover, it is possible to obtain an image with internal variety suchthat focus is placed on an arbitrary subject position, e.g., the mainperson or thing being sharp, while leaving the background solelyblurred. Hereinafter, embodiments of the imaging apparatus according tothe present invention will be described with reference to the drawings.

Embodiment 1

FIG. 1 is a schematic diagram showing an imaging apparatus A accordingto Embodiment 1. The imaging apparatus A of the present embodimentincludes: a lens optical system L having an optical axis V; an arrayoptical device K disposed near the focal point of the lens opticalsystem L; an imaging device N; a first signal processing section C1; asecond signal processing section C2; and a storage section Me.

The lens optical system L is composed of: an optical element L1 on whichlight beams B1 to B4 from a subject (not shown) are incident; a stop Son which light having passed through the optical element L1 is incident;and a lens L2 on which light having passed through the stop S isincident. The optical element L1 has an optical region D1 and an opticalregion D2, the optical region D2 having optical characteristics forproviding a different focusing characteristic from the focusingcharacteristic associated with rays having passed through the opticalregion D1. The optical element L1 may be provided near the stop S.

FIG. 2 is a front view of the optical element L1 as viewed from thesubject side. The optical region D2 has optical subregions d2A, d2B, andd2C. In the optical element L1, the optical region D1 and the opticalsubregions d2A, d2B, and d2C are four divided, upper-lower/right-leftparts around the optical axis V as a center of boundary, in a planewhich is perpendicular to the optical axis V. The optical region D1 hasidentical spectral transmittance characteristics to those of the opticalsubregion d2B. The optical region D1 and the optical subregions d2A andd2C have respectively different spectral transmittance characteristics.

For example, the optical region D1 and the optical subregion d2B havefirst spectral transmittance characteristics, such that they mainlyallow rays of the green band to pass through, while absorbing rays inany other band. The optical subregion d2A have second spectraltransmittance characteristics, such that it mainly allows rays of thered band to pass through, while absorbing rays in any other band. Theoptical subregion d1C has third spectral transmittance characteristics,such that it mainly allows rays of the blue band to pass through, whileabsorbing rays in any other band.

By providing filters which transmit light of different wavelength bands(e.g., filters made of organic materials or dielectric multilayer films)in portions of the optical element L1, for example, it can be ensuredthat the light transmitted through the respective regions of the opticalregion D1 and the optical subregions d2A, d2B, and d2C have differentwavelength bands. Alternatively, the material of the optical element L1itself may have spectral transmittance characteristics. For example, inthe case where the optical element L1 is made of a resin, coloringmaterials may be added to the resin to ensure that the wavelength bandsof light transmitted by the respective regions of the optical region D1and the optical subregions d2A, d2B, and d2C are different. In the casewhere the optical element L1 is made of glass, microparticles, e.g.,metals, may be added to the glass to ensure that the wavelength bands oflight transmitted by the respective regions are different. Instead ofusing an absorbing material, multilayered interference films may beformed on the surface of the optical element L1 to ensure that thewavelength bands of light transmitted by the respective regions aredifferent. In this case, the optical element L1 is a color filter of areflection type, which may be formed by vapor deposition technique orthe like.

In the example shown in FIG. 2, the focusing characteristics associatedwith rays transmitted through the optical subregions d2A, d2B, and d2Care identical. As a result, the same sharpness at a predeterminedsubject distance is obtained, whereby a natural color image can beobtained. However, it is not necessary for the focusing characteristicsassociated with rays transmitted through the optical subregions d2A,d2B, and d2C to be identical.

In FIG. 2, a broken line s indicates where the stop S is. The lightbeams B1, B2, B3, and B4 shown in FIG. 1 are light beams passing throughthe optical region D1 and the optical subregions d2A, d2B, and d2C ofthe optical element L1, respectively. The light beams B1, B2, B3, and B4pass through the optical element L1, the stop S, the lens L2, and thearray optical device K in this order, and arrive at the imaging plane Nion the imaging device N (shown in FIG. 4 and so on).

FIG. 3 is a perspective view of the array optical device K. On the faceof the array optical device K closer to the imaging device N, opticalelements M1 are provided in a lattice form. Each optical element M1 hascross sections (cross sections along the vertical direction and alongthe lateral direction) in arc shapes, each optical element M1 protrudingtoward the imaging device N. Thus, the optical elements M1 aremicrolenses, and the array optical device K is a microlens array.

As shown in FIG. 1, the array optical device K is provided near thefocal point of the lens optical system L, being at a position which is apredetermined distance away from the imaging plane Ni. Although theoptical characteristics of the optical element L1 will actually affectthe focusing characteristic of the entire lens optical system L, theposition of the array optical device K may be determined based on thefocal point of the lens L2 as a reference, for example.

In the present embodiment, what is meant by “different focusingcharacteristics” is that, when a comparison is made based on light of apredetermined wavelength, at least one of the characteristicscontributing to convergence of that light in the optical system isdifferent. Specifically, it is meant that, when a comparison is madebased on light of a predetermined wavelength, light having passedthrough the optical regions D1 and D2 is conducive to different focallengths of the lens optical system L, different distances from thesubject at which focus is achieved, different distance ranges in whichsharpness of a certain value or higher is obtained, and so on. Byadjusting the radii of curvature, aspheric coefficients, and refractiveindices of the optical regions D1 and D2, different focusingcharacteristics of the lens optical system L can be induced.

In the present embodiment, light having passed through the opticalregion D1 and the optical subregions d2A, d2B, and d2C through a singleinstance of imaging passes through the lens L2 and thereafter enters thearray optical device K. The array optical device K causes light havingpassed through the optical region D1 and the optical subregions d2A,d2B, and d2C to each enter a pixel(s) of the imaging device N.

FIG. 4A is an enlarged diagram showing the array optical device K andthe imaging device N shown in FIG. 1, and FIG. 4B is a diagram showingrelative positioning between the array optical device K and pixels onthe imaging device N. The array optical device K is disposed so that theface on which the optical elements M1 are formed is oriented toward theimaging plane Ni.

As shown in FIG. 4B, pixels P are disposed in a matrix shape on theimaging plane Ni. The pixels P can be classified into pixels P1, P2, P3,and P4.

For the sake of description, one set of pixels P1, P2, P3, and P4arranged in two rows by two columns is referred to as a “pixel groupPg”. In one pixel group Pg, given that the position of the pixel P1 is(1, 1), then the pixel P2 is at position (2, 1), the pixel P3 atposition (2, 2), and the pixel P4 at position (1, 2). The pixel P1 andpixel P3, both of which are entered by light of the green band, aredisposed at oblique positions in the plane of the imaging plane Ni.Thus, in the present embodiment, the wavelength bands of light enteringthe pixels P1, P2, P3, and P4 may be arranged according to a Bayerpattern. The positions of the pixel P1 and the pixel P3 may be reversed.Any arrangement other than the Bayer pattern may also be used. Sincelight of the respective wavelength bands of R, G, and B is obtained withthe optical region D1 and the optical subregions d2A, d2B, and d2C,there is no need to form color filters for the pixels P1, P2, P3, andP4, but monochromatic sensors may be used.

The array optical device K is disposed so that the face on which theoptical elements M1 are formed is oriented toward the imaging plane Ni.The array optical device K is arranged so that one optical element M1thereof corresponds to four pixels, i.e., two rows by two columns ofpixels P1 to P4 (pixel group Pg), on the imaging plane Ni. MicrolensesMs are provided on the imaging plane Ni so as to cover the surface ofthe pixels P1, P2, P3, and P4.

The array optical device K is designed so that: (a large part of) thelight beam B1 having passed through the optical region D1 (shown in FIG.1, FIG. 2) on the optical element L1 (the light beam B1 indicated bysolid lines in FIG. 1) reaches the pixel P3 on the imaging plane Ni; (alarge part of) the light beam B2 having passed through the opticalsubregion d2A reaches the pixel P4 on the imaging plane Ni; (a largepart of) the light beam B3 having passed through the optical subregiond2B reaches the pixel P1 on the imaging plane Ni; and (a large part of)the light beam B4 having passed through the optical subregion d2Creaches the pixel P2 on the imaging plane Ni. Specifically, the aboveconstruction is realized by appropriately setting parameters such as therefractive index of the array optical device K, the distance from theimaging plane Ni, and the radius of curvature of the surface of theoptical elements M1.

The filters which are used in the optical region D1 and the opticalsubregions d2A, d2B, and d2C are filters made of organic materials, forexample. Note that filters respectively having the first spectraltransmittance characteristics, the second spectral transmittancecharacteristics, and the third spectral transmittance characteristicsmainly transmit rays of mutually different wavelength bands. However,there may be partial overlaps between the wavelength bands of lighttransmitted by the respective filters. Moreover, without being limitedto primary color filters of RGB, complementary color filters (cyan,magenta, yellow) may also be used.

The stop S is a region through which light beams of all angles of viewwill pass. Therefore, by inserting a surface having opticalcharacteristics for controlling the focusing characteristic in theneighborhood of the stop S, it becomes possible to control the focusingcharacteristic of light beams of all angles of view alike. In otherwords, in the present embodiment, the optical element L1 may be providedin the neighborhood of the stop S. By disposing the optical regions D1and D2 for inducing different focusing characteristics of the opticalsystem L in the neighborhood of the stop S, the light beam can beallowed to have a focusing characteristic that is in accordance with thenumber of divided regions.

In FIG. 1, the optical element L1 is provided at a position for allowinglight having passed through the optical element L1 to be incident on thestop S directly (i.e., not via any other optical member). The opticalelement L1 may be provided on the imaging device N side of the stop S.In that case, the optical element L1 may be provided between the stop Sand the lens L2, so that light having passed through the stop S isincident on the optical element L1 directly (i.e., not via any otheroptical member).

Moreover, the array optical device K has a function of branching outinto outgoing directions depending on the incident angle of the ray.Therefore, the light beam can be branched out over the pixels on theimaging plane Ni so as to correspond to the optical region D1 and theoptical subregions d2A, d2B, and d2C as divided near the stop S.

The first signal processing section C1 (shown in FIG. 1) generates acolor image by using a plurality of pieces of luminance informationobtained from the pixels P1, P2, P3, and P4 through a single instance ofimaging. Hereinafter, the specific method of color image generation willbe described.

In the optical system of the imaging apparatus A in FIG. 1, the opticalregion D1 has a non-spherical surface, whereas the optical region D2(the optical subregions d2A, d2B, and d2C) has a planar surface. Forsimplicity of description, it is assumed that the lens L2 is an ideallens free of aberration.

Since the optical region D2 has a planar surface, rays having passedthrough the optical region D2 and the lens L2 have no (or little)spherical aberration, as in the graph indicated by a solid line in FIG.5. When there is no spherical aberration, the point spread distributionvaries with an increase in shift from the focal point. In other words,the point spread distribution varies with changing subject distance.

Moreover, due to the aspherical shape of the optical region D1, there isspherical aberration associated with rays having passed through theoptical region D1 and the lens L2 as shown by the graph indicated by abroken line in FIG. 5. Such spherical aberration can be imparted byadjusting the aspherical shape of the optical region D1. With suchspherical aberration, in a predetermined range near the focal point ofthe lens optical system L, the point spread distribution associated withrays having passed through the optical region D1 can be keptsubstantially constant. In other words, the point spread distributioncan be kept substantially constant within the predetermined subjectdistance range.

FIG. 6 is a conceptual diagram of point spread distributions fordifferent subject distances. The left (as one faces the figure) columnin FIG. 6 shows what is obtained by extracting only the point spreaddistribution of the pixel P3 while masking the point spreaddistributions of the pixels P1, P2, and P4 at 0 level. In other words,it is a point spread distribution which is created by a light beamhaving passed through the optical region D1. Moreover, the right columnis obtained by extracting only the point spread distribution of thepixel P1, while masking the point spread distributions of the pixels P2,P3, and P4 at 0 level. In other words, it is a point spread distributionwhich is created by a light beam having passed through the opticalsubregion d2B. It can be seen that the point spread distribution of thepixel P3 is substantially constant against changing subject distance,and that the point spread distribution of the pixel P1 has its pointimage decrease in size as the subject distance increases.

Sharpness also changes with changes in point spread distribution. Sincethe image sharpness increases as the point image decreases in size, agraph indication of the relationship between subject distance andsharpness will result in a relationship as shown in FIG. 7. In the graphof FIG. 7, G1 represents the sharpness in a predetermined region of animage formed at the pixel P3 (green component)(an image formed by lighthaving passed through the optical region D1), and G2, R, and Brespectively represent the sharpnesses in a predetermined region ofimages formed at the pixel P1 (green component), the pixel P4 (redcomponent), and P2 (blue component).

Sharpness can be determined based on differences between the luminancevalues of adjacent pixels in an image block of a predetermined size.Alternatively, it may be determined based on a frequency spectrumobtained by applying Fourier transform to the luminance distribution ofan image block of a predetermined size.

When determining a sharpness E in a block of a predetermined size foreach component of the pixels P1, P2, P3, and P4 based on differencesbetween the luminance values of adjacent pixels, (math. 1) is used, forexample.

$\begin{matrix}{E = {\sum\limits_{i}{\sum\limits_{j}\;\sqrt{\left( {\Delta\; x_{i,j}} \right)^{2} + \left( {\Delta\; y_{i,j}} \right)^{2}}}}} & \left\lbrack {{math}.\mspace{14mu} 1} \right\rbrack\end{matrix}$

Since the pixels P1, P2, P3, and P4 compose a Bayer pattern as mentionedearlier, the sharpness of each component is to be determined through acalculation by extracting pixel information from every other pixel alongboth the x direction and the y direction of the image.

In (math. 1), Δx_(i,j) is a difference value between the luminance valueof a pixel at coordinates (i,j) and the luminance value of a pixel atcoordinates (i+2,j) within an image block of a predetermined size; andΔy_(i,j) is a difference value between the luminance value of a pixel atcoordinates (i,j) and the luminance value of a pixel at coordinates(i,j+2), within the image block of the predetermined size. The reasonwhy the calculation is done by using coordinate j and coordinate j+2 isthat, in an image obtained at each of the pixels P3 and P1, luminanceinformation along the lateral direction (x direction) or the verticaldirection (y direction) is created for every other pixel.

From the calculation of (math. 1), the greater the difference betweenluminance values in the image block of the predetermined size is, thegreater sharpness is obtained.

Although image sharpness can be determined by using (math. 1) mentionedabove, it may also be determined based on a frequency spectrum obtainedby applying Fourier transform to the sharpness within the block of thepredetermined size.

FIGS. 8( a) to (c) each show a luminance distribution in an image blocksized 16×16. Sharpness decreases in the order of FIGS. 8( a), (b), (c).FIGS. 8( d) to (f) show frequency spectrums obtained by applying atwo-dimensional Fourier transform to the respective image blocks shownin FIGS. 8( a) to (c). In FIGS. 8( d) to (f), for ease of understanding,the intensity of each frequency spectrum is indicated throughlogarithmic transformation, such that the more intense the frequencyspectrum is, the brighter it appears. In each frequency spectrum, theplace of highest luminance in the center is a DC component, andincreasingly higher frequencies exist closer to the peripheral portion.

In FIGS. 8( d) to (f), it can be seen that the higher frequency spectrumis lost as the image sharpness decreases. Therefore, in order todetermine sharpness from any such frequency spectrum, the entirefrequency spectrum or a portion thereof may be extracted, for example.

When generating a color image, the color image may be generated bysimply interpolating the chromatic information that is lost for eachpixel position on the basis of the luminance information of the pixelsP1, P2, P3, and P4; however, the sharpness of G1 is smaller than thesharpnesses of G2, B, and R as shown in FIG. 7, and therefore the colorimage may be generated after enhancing the sharpness of G1.

FIGS. 9A to 9E is a diagram describing a method of enhancing thesharpness of G1 based on the sharpnesses of G2, B, and R. FIG. 9A showsa subject, which is a white-black chart, and FIG. 9B is a diagramshowing a cross section in the luminance of the subject of FIG. 9A. Asshown in FIG. 9B, the luminance of the chart has a step-like crosssection; however, the image will have a luminance cross section as shownin FIG. 9C when taken by placing the chart at a predetermined positionthat is shifted slightly frontward from the subject position at whichthe rays reaching the pixels P1, P2, and P4 are best focused, forexample. In the graph of FIG. 9C, G1 is a luminance cross section of animage which is generated at the pixel P3 (green component), whereas G2,B, and R are luminance cross sections of images which are generated atthe pixel P1 (green component), the pixel P2 (blue component), and thepixel P4 (red component), respectively. Thus, the luminance crosssections of G2, B, and R is closer to the luminance cross section of theactual chart in FIG. 9B than is the luminance cross section of G1,therefore having a higher sharpness.

When a white-black chart such as that shown in FIG. 9A is imaged, therespective luminance cross sections of G2, B, and R will havesubstantially identical cross sections; in actuality, however, a subjectimage of every possible color component will be taken, and the luminancecross sections of G2, B, and R in FIG. 9C will not coincide in mostcases. Therefore, the respective sharpnesses may be detected from theluminance cross sections of G2, B, and R, and a color component with ahigh sharpness may be selected to sharpen the luminance cross section ofG1. Detection of sharpness is performed at a sharpness detection sectionwhich is in the first signal processing section C1. When a luminancecross section with a high sharpness is selected, and its luminance crosssection is subjected to second-order differentiation, the distributionof FIG. 9D is obtained, and the edge of an image of the color componentwith a high sharpness can be detected. Next, by subtracting thedistribution of FIG. 9D from the G1 luminance distribution of FIG. 9C,the distribution of FIG. 9E is obtained, whereby the G1 luminancedistribution has been sharpened. Now, when subtracting the distributionof FIG. 9D, the distribution of FIG. 9D may be multiplied by apredetermined coefficient, which then may be subtracted from the G1luminance distribution of FIG. 9C, thus controlling the degree ofsharpening G1.

Although the present embodiment illustrates the image sharpening inone-dimensional terms for simplicity of description, an image istwo-dimensional and therefore a two-dimensional sharpening process isactually to take place.

Through the above image processing, the sharpness of G1 which isindicated by a solid line in FIG. 7 can be sharpened as in G1′ which isindicated by a broken line, thus sharpening the resultant color image.

FIG. 10 is a graph showing the relationship between subject distance andsharpness in the case where the optical surface in the optical region D1is changed from an aspherical shape to a spherical shape in FIG. 1. Inthis case, too, the color image can be sharpened similarly to FIG. 7.

In FIG. 10, different color components have a high sharpness dependingon the subject distance. Therefore, respective sharpnesses are detectedfrom the luminance cross sections of G1, G2, R, and B, and the colorcomponent with the highest sharpness is selected to sharpen any othercolor component.

Through the above image processing, the sharpnesses of G1, G2, R, and Bwhich are indicated by solid lines in FIG. 10 can be respectivelysharpened as in G1′, G2′, R′, and B′ which are indicated by brokenlines, thus sharpening the resultant color image.

Next, another image sharpening technique will be described. FIG. 11 is adiagram describing a method of enhancing the sharpnesses of G2, B, and Rbased on G1′, which is a sharpness-enhanced version of G1. Theconstruction of the optical regions D1 and D2 is the same as that inFIG. 7, and the point spread distribution created by rays having passedthrough the optical region D1 is substantially constant within apredetermined subject distance range. Therefore, the point spreaddistribution which is created by extracting each pixel P3 (G1 component)is substantially constant within a predetermined subject distance range.So long as the point spread distribution is substantially constant inthe predetermined subject distance range, an image which is formed byextracting the pixel P3 (G1 component) is restorable based on apredetermined point spread distribution, regardless of the subjectdistance.

Hereinafter, a method of restoring a captured image based on apreviously stored point spread function will be described. Assuming thatthe original image is f(x,y), and the point spread distribution ish(x,y), the captured image g(x,y) is expressed by (math. 2).g(x,y)=f(x,y)

h(x,y) (where

represents convolution)  [math. 2]

A Fourier transform applied to both sides of (math. 2) gives (math. 3).G(u,v)=R(u,v)H(u,v)  [math. 3]

Now, by applying an inverse filter Hinv(u,v) of (math. 4) to thedeteriorated image G(u,v), a two-dimensional Fourier transform F(u,v) ofthe original image is obtained as in (math. 5). By applying an inverseFourier transform to this, the original image f(x,y) can be obtained asa restored image.

$\begin{matrix}{{{Hinv}\left( {u,v} \right)} = \frac{1}{H\left( {u,v} \right)}} & \left\lbrack {{math}.\mspace{14mu} 4} \right\rbrack \\{{F\left( {u,v} \right)} = {{{Hinv}\left( {u,v} \right)}{G\left( {u,v} \right)}}} & \left\lbrack {{math}.\mspace{14mu} 5} \right\rbrack\end{matrix}$

However, if H(u,v) is 0 or has a very small value, Hinv(u,v) willdiverge; therefore, a Wiener filter Hw(u,v) as indicated by (math. 6) isused for restoration from the deteriorated image.

$\begin{matrix}{{{Hw}\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2}}{{{H\left( {u,v} \right)}}^{2} + {{{N\left( {u,v} \right)}}^{2}/{{F\left( {u,v} \right)}}^{2}}}}} & \left\lbrack {{math}.\mspace{14mu} 6} \right\rbrack\end{matrix}$

In (math. 6), N(u,v) is noise. Since usually the noise and the originalimage F(u,v) are unknown, a constant k is actually used to restore thedeteriorated image with a filter of (math. 7).

$\begin{matrix}{{{Hw}\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2}}{{{H\left( {u,v} \right)}}^{2} + k}}} & \left\lbrack {{math}.\mspace{14mu} 7} \right\rbrack\end{matrix}$

With such a restoration filter, the sharpness of G1 which is indicatedby a solid line in FIG. 11 can be sharpened as in G1′ which is indicatedby a dotted line. Thus, according to the present embodiment, by using apoint spread function, optical it is possible to perform a restorationprocess for all regions of an image which is formed from luminanceinformation of the pixels that are reached by light entering the regionD1. Since a point spread function generally changes with the imagingposition of the optical system, a point spread function corresponding toeach imaging position may be used. However, in an optical system whosepoint spread function hardly depends on the imaging position, it ispossible to perform a restoration process for all regions of an imagewith a single point spread function. While there is a need to store thepoint spread function in advance in a memory or the like, use of asingle point spread distribution allows the amount of used memory to bereduced. Furthermore, in a manner similar to the method shown in FIG. 9,from the G1′ luminance cross section (restored sharpened image),respective sharpnesses may be detected for each predetermined region(sharpness detection section); the luminance cross section of a colorcomponent with a high sharpness may be subjected to second-orderdifferentiation; and this may be subtracted from G2, B, and R, wherebythe sharpnesses of G2, B, and R can be improved as in G2′, B′, and R′which is indicated by a broken line in FIG. 11. Thus, the luminanceinformation components of other pixels can be sharpened based onsharpness.

Through the above image processing, the sharpness of G1 and thesharpnesses of G2, B, and R which are indicated by solid lines in FIG.11 are improved as in G1′ which is indicated by a dotted line and G2′,B′, and R′ which are indicated by a broken line, thus sharpening theresultant color image. Through such a sharpening process, the depth offield can be further expanded from the sharpening process described inFIG. 7.

Next, a specific method of determining subject distance will bedescribed.

FIG. 12 is a cross-sectional view showing the imaging apparatus A ofEmbodiment 1. In FIG. 12, constituent elements which are identical tothose in FIG. 1 are indicated by the same alphanumeric expressions as inFIG. 1. Although FIG. 12 omits the array optical device K (shown in FIG.1 and the like) from illustration, the array optical device K isactually included in a region H in FIG. 12. The region H has theconstruction shown in FIG. 4A.

Table 1 and Table 2 show design data for the optical system of theimaging apparatus A shown in FIG. 12. In Table 1 and Table 2, Rirepresents a paraxial radius of curvature (mm) of each surface; direpresents an inter-surface-center interval (mm) of each surface; ndrepresents a d-line refractive index of the lens or filter; and v drepresents a d-line Abbe number of each optical element. Moreover, anaspherical shape is expressed by (math. 8), where x is a distance from atangent plane of the surface vertex in the optical axis direction; h isa height from the optical axis; r is a paraxial radius of curvature; kis a conical constant; and Am (m=4,6,8,10) is an m^(th) asphericcoefficient. Moreover, Table 3 shows spectral transmittancecharacteristics of the optical region D1 and the optical subregions d2A,d2B, and d2C. The optical region D1 and the optical subregion d2B haveidentical spectral transmittance characteristics.

$\begin{matrix}{x = {\frac{\frac{1}{r}h^{2}}{1 + \sqrt{1 - {\left( {1 + k} \right)\left( \frac{1}{r} \right)^{2}h^{2}}}} + {A_{4}h^{4}} + {A_{6}h^{6}} + {A_{8}h^{8}} + {A_{10}h^{10}}}} & \left\lbrack {{math}.\mspace{14mu} 8} \right\rbrack\end{matrix}$

TABLE 1 lens data focal length = 10 mm, F value = 2.8 angle of view 2ω =10°, effective imaging circle = φ1.75 mm surface number Ri di di νdobject ∞ 4000 — — R1 surface region ∞ 0.5 1.5253 56.0 d2A region 2000 D1region ∞ d2C region ∞ d2B R2 surface ∞ 0.1 — — stop ∞ 10 — — R3 surface5.332237 5 1.5253 56.0 R4 surface −319.8501 6.75 — — image ∞ — — —surface

TABLE 2 k A4 A6 A8 A10 R1 region 0 0 0 0 0 surface D2 region 00.000064653 0.00018193 0 0 D1 R3 surface −0.296987 0.000421138−0.000059237 0.000016321 −0.000001294 R4 surface 0 0.00274336−0.000566209 0.000216386 −0.000026416

TABLE 3 spectral transmittance characteristics region d2A 400 to 500 nmregion D1 500 to 600 nm region d2C 600 to 700 nm region d2B 500 to 600nm

The first signal processing section C1 shown in FIG. 1 outputs a firstimage I1 (shown in FIG. 1) which is obtained by extracting luminanceinformation from the pixel P3 (G1 component) and a second image I2 whichis obtained by extracting luminance information from the pixel P1 (G2component). Since the two optical regions D1 and D2 have mutuallydifferent optical characteristics, the image sharpnesses (valuescalculated by using luminance) of the first and second images I1 and I2vary depending on the subject distance. In the storage section Me (shownin FIG. 1), a correlation between sharpness and subject distance oflight having passed through each of the optical regions D1 and D2 isstored. In the second signal processing section C2 (shown in FIG. 1),the distance from the subject can be determined based on the sharpnessesof the first and second images I1 and I2 and the aforementionedcorrelations.

Herein, the range Z in FIG. 7 and FIG. 11 represents a region in whichG2 changes but in which G1 hardly changes. In the range Z, the subjectdistance can be determined by utilizing this relationship. For example,in the range Z, since there is a correlation between the subjectdistance and the ratio between the sharpnesses G1 and G2, thecorrelation between the subject distance and the ratio between thesharpnesses G1 and G2 is stored in the storage section Me in advance.

When the imaging apparatus is used, within the data which is obtainedthrough a single instance of imaging, a ratio in sharpness between thefirst image I1 which is generated from the pixel P3 (G1 component) aloneand the second image I2 which is generated from the pixel P1 (G2component) alone is determined for each calculation block. Then, byusing the correlations stored in the storage section Me, the subjectdistance can be determined. Specifically, for each calculation block,the sharpness ratio in the aforementioned correlation and the sharpnessratio between the first image I1 and the second image I2 are compared invalue. Then, a subject distance that corresponds to a matching valuebetween the two is regarded as the distance from the subject atshooting.

In order to univocally determine the subject distance from the ratiobetween the sharpness of the first image I1 generated from the pixel P3alone and the sharpness of the second image I2 generated from the pixelP1 alone, it is necessary that the sharpness ratio always variesthroughout a predetermined subject distance range.

In FIG. 7, FIG. 10, and FIG. 11, the sharpness ratio always variesthroughout the range Z, and thus the subject distance can be univocallydetermined. Moreover, since the sharpness values being too low wouldmake it impossible to determine a ratio, the sharpness values may beequal to or greater than a certain value.

Note that the relationship between subject distance and sharpness isdetermined from the radii of curvature, aspheric coefficients, andrefractive indices of the surfaces in the optical regions D1 and D2. Inother words, the optical regions D1 and D2 need to have opticalcharacteristics such that the ratio between the sharpness of the firstimage I1 and the sharpness of the second image I2 always variesthroughout the predetermined distance range.

In the present embodiment, the subject distance may be determined byusing any value other than sharpness, e.g., contrast, so long as it is avalue that is calculated by using luminance (luminance information).Contrast can be determined from a ratio between the highest luminancevalue and the lowest luminance value within a predetermined calculationblock, for example. Sharpness is a difference between luminance values,whereas contrast is a ratio between luminance values. A contrast may bedetermined from a ratio between a point of the highest luminance valueand a point of the lowest luminance value, or a contrast may bedetermined from a ratio between an average value of several points ofthe largest luminance values and an average value of several points ofthe lowest luminance values. When the subject distance is within acertain range, the contrast of the first image I1 and the contrast ofthe second image I2 have a correlation with the subject distance. In thecase of using contrast to determine the subject distance, similarly tothe case of sharpness, a correlation between the subject distance andthe contrast ratio is stored in advance in the storage section Me. Inthis case, the first signal processing section C1 includes a contrastdetection section which detects the contrast of the first image I1obtained from the pixel P3 and the contrast of the second image I2obtained from the pixel P1. By determining a contrast ratio between thefirst image I1 and the second image I2 for each calculation block, it ispossible to determine the subject distance by utilizing the correlation(second signal processing section C2).

Moreover, in the present embodiment, the subject distance may bedetermined by using a value other than sharpness or contrast, e.g.,point spread distribution. Hereinafter, a method of determining a pointspread distribution from the first image I1 and the second image I2 willbe described.

When the aforementioned (math. 7) is used to restore the first image I1generated from the pixel P3 (G1 component) alone, a restored imagei1′(x,y) which is very close to the original image f(x,y) is obtained.Now, assuming a second image i2(x,y) which is generated from the pixelP1 (G2 component) alone, and a point spread distribution h2(x,y)associated with rays passing through the optical region D2, it ispossible to express i2(x,y) by (math. 9).i2(x,y)≈I1′(x,y)

h2(x,y) (where

represents convolution)  [math. 9]

A Fourier transform applied to both sides of (math. 9) gives (math. 10).I2(u,v)≈I1′(u,v)H2(u,v)  [math. 10]

Through transformation of (math. 10), frequency domain values H2(u,v) ofthe point spread distribution h2(x,y) are obtained as in (math. 11).

$\begin{matrix}{{H\; 2\left( {u,v} \right)} \approx \frac{I\; 2\left( {u,v} \right)}{I\; 1^{\prime}\left( {u,v} \right)}} & \left\lbrack {{math}.\mspace{14mu} 11} \right\rbrack\end{matrix}$

By applying an inverse Fourier transform to this, a point spreaddistribution h2(x,y) associated with rays passing through the opticalregion D2 can be obtained.

Since the point spread distribution h2(x,y) associated with rays passingthrough the optical region D2 varies with subject distance, when thesubject distance is within a certain range, the point spreaddistribution h2(x,y) and the subject distance have a correlation. Thiscorrelation can be utilized to determine the subject distance.

In the case of using a representative value to express a point spreaddistribution, the diameter of the point spread distribution can be used,for example. Similarly to the case of sharpness or contrast, acorrelation between subject distance and point image diameter is storedin advance in the storage section Me. By determining a point spreaddistribution from the first image I1 or the second image I2 for eachblock, and determining the diameter of the point image from the pointspread distribution, the subject distance can be determined throughcorrelation. The diameter of a point image can be determined from thehalf-width of the point spread distribution, for example.

The present embodiment may be constructed so as to generate an imageobtained through addition of the first image I1 and the second image I2in the case where the radii of curvature of the respective regions aremade different from each other as shown in FIG. 10. The distance rangein which sharpness attains a certain value or greater in the imagegenerated through addition of the first image I1 and the second image I2is larger than those of the first image I1 and the second image I2. Inthis case, the ratio between the sharpness of the image generatedthrough addition and the sharpness of either the first image I1 or thesecond image I2 has a correlation with subject distance. By storing thiscorrelation in advance, it is possible to determine a subject distancefor each predetermined region of an image.

Note that the optical system of the imaging apparatus of the presentembodiment may be an image-side telecentric optical system. As a result,even if the angle of view changes, incidence occurs with theprincipal-ray incident angle of the array optical device K having avalue close to 0 degrees, so that the crosstalk between light beamsreaching the pixels P1, P2, P3, and P4 can be reduced across the entireimaging region.

Although the present embodiment has illustrated the lens L2 to be anideal lens for simplicity of description as mentioned above, it is notnecessary to employ an ideal lens. For example, a non-ideal lens wouldhave axial chromatic aberration, but it is possible to select a colorcomponent with a high sharpness to sharpen other color components asdescribed earlier; thus, a color image with sharpness can be generatedeven without an ideal lens. Moreover, in the case of determining thesubject distance, the distance is to be determined based on a singlecolor component (which in the present embodiment is the greencomponent); thus, there may be some axial chromatic aberration.

Moreover, in the case of using a non-ideal lens, a construction thatcorrects for axial chromatic aberration at the optical element L1 may beemployed. Although the present embodiment assumes that the opticalregion D2 (the optical subregions d2A, d2B, and d2C) of the opticalelement L1 is all planar surface, they may respectively have differentoptical surfaces to correct for axial chromatic aberration. As describedearlier, rays having passed through the optical subregions d2A, d2B, andd2C reach the pixel P4, the pixel P1, and the pixel P2, respectively.Light of the red, green, and blue wavelength components mainly reach thepixel P4, the pixel P1, and the pixel P2, respectively; therefore, inthe case where a lens having axial chromatic aberration is adopted forthe lens L2, the optical surfaces of the optical subregions d2A, d2B,and d2C may be allowed to have different optical powers so that therespective wavelength bands of light have the same focusing position.With such a construction, as compared to the case where the opticalsubregions d2A, d2B, and d2C have an equal optical power, the focusingpositions of light transmitted through the optical subregions d2A, d2B,and d2C can be brought close to one another, whereby the axial chromaticaberration occurring in the lens L2 can be corrected for by the opticalelement L1. By correcting for the axial chromatic aberration with theoptical element L1, the number of lenses composing the lens L2 can bereduced, thus downsizing the optical system.

Although the optical element L1 and the lens L2 are separate in thepresent embodiment, another possible construction is where the lens L2has the optical regions D1 and D2, with the optical element L1 beingeliminated. In this case, the stop S may be disposed near the opticalregions D1 and D2 of the lens L2.

In this Embodiment 1, the optical region D1 and the optical subregiond2B are equal in area. With this construction, the exposure times forthe pixel P3 and the pixel P1 may be made equal. In the case where theoptical region D1 and the optical subregion d2B have different areas,the exposure times for the pixel P3 and the pixel P1 may be different.For example, when the area of the optical region D1 is broader than thearea of the optical subregion d2B, the exposure time for the pixel P3may be shorter than the exposure time for the pixel P1.

Thus, according to the present embodiment, both a color image and thesubject distance can be obtained through a single instance of imagingusing a single imaging system. In other words, through a single instanceof imaging using the imaging apparatus of the present embodiment,luminance information for a color image output and subject distancemeasurement can be obtained. Then, using the luminance information, boththe color image and the subject distance can be obtained. Since thesubject distance can be calculated for each calculation block, it ispossible to obtain the subject distance at any arbitrary image positionin the color image. Thus, it is also possible to obtain a subjectdistance map across the entire image. Moreover, the distance from thesubject can be obtained with a single imaging system, which is unlike inan imaging apparatus having a plurality of imaging optical systems,where it would be necessary to ensure matching characteristics andpositions between the plurality of imaging optical systems. Moreover,since rays enter the optical regions D1 and D2 (optical subregions d2A,d2B, and d2C) through a single instance of imaging, discrepanciesbetween images will not occur due to time lags in imaging. Moreover,when a motion video is shot by using the imaging apparatus of thepresent embodiment, an accurate distance from the subject can bemeasured even if the subject position changes with lapse of time.

In the present embodiment, the imaging apparatus may have a constructionas shown in FIG. 13. FIG. 13 is a diagram showing another imagingapparatus according to Embodiment 1. In the imaging apparatus shown inFIG. 13, the first signal processing section C1 outputs a first image I1obtained from the pixel P3, a second image I2 obtained from the pixelP1, and third and fourth images I3 and 14 obtained from the pixels P2and P4. A second signal processing section C2 performs a distancemeasurement calculation by using luminance information which isexpressed as a difference (sharpness) in luminance value betweenadjacent pixels in the first image I1 and the second image I2. A thirdsignal processing section C3 generates a color image by merging at leasttwo or more images of different wavelength bands from among the imagesI1 to I4 of respective wavelength bands.

In the construction shown in FIG. 13, a color image is formed by mergingthe images I2, I3, and I4, for example. Although it would be possible touse the image I1 instead of the image I2, this in itself would result inan unnatural image because the sharpness G1 for the subject distance isdifferent from G2, G3, or G4, as shown in FIG. 7. Therefore, in the caseof using the image I1 for color image formation, a conversion into aclear image through image processing may be performed as indicated inFIG. 7. For example, a sharpening process through a Laplacian filterprocess may be employed.

Table 1 assumes that the three optical subregions d2A, d2B, and d2C haveplanar surfaces while the optical region D1 has a non-spherical surfacegenerating a substantially constant point spread distribution.Alternatively, as shown in FIG. 14, the optical region D1 may have aplanar surface, whereas the three optical subregions d2A, d2B, and d2Cmay have optical surfaces generating a substantially constant pointspread distribution. In this case, similarly to FIG. 2, a distance fromthe subject can be measured by using the images I1 and I2 having passedthrough the optical region D1 and the optical subregion d2B. Moreover, acolor image can be generated by merging the images obtained from therespective pixels. At this time, any image with a low sharpness may besharpened through image processing, so that a clear color image will beobtained.

Moreover, as shown in FIG. 15, the optical region D1 and the opticalsubregion d2C may have planar surfaces, whereas the two opticalsubregions d2A and d2B may have optical surfaces generating asubstantially constant point spread distribution. In this case, it maydo well to apply a sharpening process to the image I4 and the image I2before color image formation.

Embodiment 2

Embodiment 2 differs from Embodiment 1 in that a microlens array isformed on the imaging plane. In the present embodiment, any detaileddescription directed to similar subject matter to Embodiment 1 will beomitted.

FIGS. 16A and 16B are enlarged diagrams each showing an array opticaldevice K and an imaging device N. In the present embodiment, themicrolens array Md is formed on an imaging plane Ni of the imagingdevice N. Similarly to Embodiment 1 and the like, pixels P are disposedin a matrix shape on the imaging plane Ni. A single optical element of amicrolens corresponds to the plurality of pixels P. In the presentembodiment, too, similarly to Embodiment 1, light beams having passedthrough different regions of the optical element L1 can be led torespectively different pixels. FIG. 16B is a diagram showing a variantof the present embodiment. In the construction shown in FIG. 16B,microlenses Ms are disposed on the imaging plane Ni so as to cover thepixels P, and the array optical device K is stacked on the surface ofthe microlens Ms. With the construction shown in FIG. 16B, theefficiency of convergence can be enhanced over that of the constructionin FIG. 16A.

Embodiment 3

This Embodiment 3 differs from Embodiments 1 and 2 in that alight-shielding member is provided at the boundaries between opticalregions of the optical element L1. In the present embodiment, anydetailed description directed to similar subject matter to Embodiment 1will be omitted.

FIG. 17A is a front view in which a light-shielding member Q is providedat the boundaries between optical regions D1 and D2 according toEmbodiment 3. FIG. 17B is a front view in which a light-shielding memberQ is provided at the boundaries between an optical region D1 and opticalsubregions d2A, d2B, and d2C according to Embodiment 3.

At the boundaries between regions, discontinuous changes in shape causelevel differences at the boundaries, possibly causing unwanted light.Therefore, disposing the light-shielding member Q at the boundaries cansuppress unwanted light. As the light-shielding member Q, a polyesterfilm with carbon black knead therein or the like may be used, forexample. The light-shielding member Q may be formed so as to be integralwith the stop.

FIG. 17B shows an implementation in which the linear light-shieldingmember Q is employed in such a manner that the shapes of the portionspartitioned by the light-shielding member Q appear as sectors of acircle. In the present embodiment, a light-shielding member may be usedwhose light-transmitting portions are apertures in the shapes ofcircles, ellipses, rectangles, etc., so that circles, ellipses, orrectangles are obtained as the portions partitioned by thelight-shielding member Q.

Embodiment 4

FIG. 18 is a schematic diagram showing an imaging apparatus A accordingto Embodiment 4. The imaging apparatus A of the present embodimentincludes: a lens optical system L having an optical axis V; an arrayoptical device K disposed near the focal point of the lens opticalsystem L; an imaging device N; a second signal processing section C2; athird signal processing section C3; and a storage section Me.

The lens optical system L is composed of: an optical element L1including two optical regions D1 and D2 having optical characteristicswhich provide mutually different focusing characteristics, and on whichlight beams B1 and B2 from a subject (not shown) are incident; a stop Son which light having passed through the optical element L1 is incident;and a lens L2 on which light having passed through the stop S isincident. The optical element L1 may be provided near the stop S.

FIG. 19 is a front view of the optical element L1 as viewed from thesubject side. In the optical element L1, the optical regions D1 and D2are two divided, upper-lower parts around the optical axis V as a centerof boundary, in a plane which is perpendicular to the optical axis V. InFIG. 19, a broken line s indicates where the stop S is. In FIG. 18, thelight beam B1 is a light beam passing through the optical region D1 onthe optical element L1, and the light beam B2 is a light beam passingthrough the optical region D2 on the optical element L1. The light beamsB1 and B2 pass through the optical element L1, the stop S, the lens L2,and the array optical device K in this order, and arrive at the imagingplane Ni on the imaging device N (shown in FIG. 21 and so on).

FIG. 20 is a perspective view of the array optical device K. On the faceof the array optical device K closer to the imaging device N, aplurality of optical elements M1, each longitudinally extending in thelateral direction, are flanking along the vertical direction in a planewhich is perpendicular to the optical axis V. The cross section (a crosssection along the vertical direction) of each optical element M1 has anarc shape protruding toward the imaging device N. Thus, the arrayoptical device K has a lenticular lens construction.

As shown in FIG. 18, the array optical device K is provided near thefocal point of the lens optical system L, being at a position which is apredetermined distance away from the imaging plane Ni. Although theoptical characteristics of the optical element L1 will actually affectthe focusing characteristic of the entire lens optical system L, theposition of the array optical device K may be determined based on thefocal point of the lens L2 as a reference, for example. In the presentembodiment, what is meant by “different focusing characteristics” isthat at least one of the characteristics contributing to lightconvergence in the optical system is different, specifically: differentfocal lengths, different distances from the subject at which focus isachieved, different distance ranges in which sharpness of a certainvalue or higher is obtained, and so on. By adjusting the radii ofcurvature, aspheric coefficients, and refractive indices of the opticalregions D1 and D2, different focusing characteristics of the lensoptical system L can be induced.

In the present embodiment, light having passed through the two opticalregions D1 and D2 passes through the lens L2 and thereafter enters thearray optical device K. The array optical device K causes light havingpassed through the optical region D1 to enter the pixel P1 (shown inFIG. 21 and so on) of the imaging device N and light having passedthrough the optical region D2 to enter the pixel P2 of the imagingdevice N.

FIG. 21A is an enlarged diagram showing the array optical device K andthe imaging device N shown in FIG. 18, and FIG. 21B is a diagram showingrelative positioning between the array optical device K and pixels onthe imaging device N. The array optical device K is disposed so that theface on which the optical elements M1 are formed is oriented toward theimaging plane Ni. Pixels P are disposed in a matrix shape on the imagingplane Ni. The pixels P can be classified into pixels P1 and P2.

The pixels P1 are arranged in one row along the lateral direction (rowdirection). Along the vertical direction (column direction), pixels P1are arranged in every other row. Moreover, the pixels P2 are arranged inone row along the lateral direction (row direction). Along the verticaldirection (column direction), pixels P2 are arranged in every other row.Moreover, rows of pixels P1 and rows of pixels P2 alternate along thevertical direction (column direction).

The array optical device K is arranged so that one optical element M1thereof corresponds to two rows of pixels, consisting of one row ofpixels P1 and one row of pixels P2, on the imaging plane Ni. MicrolensesMs are provided on the imaging plane Ni so as to cover the surface ofthe pixels P1 and P2.

The array optical device K is designed so that: a large part of thelight beam B1 having passed through the optical region D1 (shown in FIG.18, FIG. 19) on the optical element L1 (the light beam B1 indicated bysolid lines in FIG. 18) reaches the pixel P1 on the imaging plane Ni;and a large part of the light beam having passed through the opticalregion D2 (the light beam B2 indicated by broken lines in FIG. 18)reaches the pixel P2 on the imaging plane Ni. Specifically, the aboveconstruction is realized by appropriately setting parameters such as therefractive index of the array optical device K, the distance from theimaging plane Ni, and the radius of curvature of the surface of theoptical elements M1.

The stop S is a region through which light beams of all angles of viewwill pass. Therefore, by inserting a surface having opticalcharacteristics for controlling the focusing characteristic in theneighborhood of the stop S, it becomes possible to control the focusingcharacteristic of light beams of all angles of view alike. In otherwords, in the present embodiment, the optical element L1 may be providedin the neighborhood of the stop S. By disposing the optical regions D1and D2 having optical characteristics which provide mutually differentfocusing characteristics in the neighborhood of the stop S, the lightbeam can be allowed to have a focusing characteristic that is inaccordance with the number of divided regions.

In FIG. 18, the optical element L1 is provided at a position forallowing light having passed through the optical element L1 to beincident on the stop S directly (i.e., not via any other opticalmember). The optical element L1 may be provided on the imaging device Nside of the stop S. In that case, the optical element L1 may be providedbetween the stop S and the lens L2, so that light having passed throughthe stop S is incident on the optical element L1 directly (i.e., not viaany other optical member).

Moreover, the array optical device K has a function of branching outinto outgoing directions depending on the incident angle of the ray.Therefore, the light beam can be branched out over the pixels on theimaging plane Ni so as to correspond to the optical regions D1 and D2 asdivided near the stop S.

FIG. 22 is a flowchart describing processing by the signal processingsections according to the present embodiment. A signal processingsection has a function of generating a refocused image. As used herein,refocusing means, by using an image which is obtained with an imagingapparatus (captured image), reconstructing an image which is focused ona subject which is at a desired (arbitrary) subject distance. Note thata “subject distance” means the distance from the imaging apparatus to asubject. Through refocusing, in an image, the sharpness of a subjectthat is at a desired subject distance becomes higher than the sharpnessof the surrounding region. A refocused image is an image in which thesharpness of a subject at a desired subject distance is higher than thesharpness of the surrounding region.

As shown in FIG. 22, at step ST1, first, luminance information of animage obtained from the imaging device N is acquired, and the image issharpened as necessary. Herein, specific examples of “luminanceinformation” are sharpness, contrast, and point spread distribution. Asshown in FIG. 18, images to be obtained from the imaging device N may beeither a first image I1 from the first pixel P1 or a second image I2from the second pixel P2. In this step ST1, the luminance information ofthese two images I1 and 12 is acquired.

Next, at step ST2, by using the luminance information, a distance fromthe subject is calculated for each predetermined region in the image,thus generating a depth map.

Next, at step ST3, based on the position at which focus is desired (bestfocus position), PSF is generated for each subject position. The bestfocus position may be externally input by a user to the imagingapparatus A, or may be determined by the second signal processingsection C2 in the imaging apparatus A.

Finally, at step ST4, the PSFs which are determined based on the depthmap are convoluted into the sharpened image, thus generating a refocusedimage at the arbitrary position. For example, steps ST1 to ST3 areperformed by the second signal processing section C2, whereas step ST4is performed by the third signal processing section C3. Note that theimage sharpening step of step ST1 and steps ST2 and ST3 may be reversedas necessary. Hereinafter, each item of the flowchart will bespecifically described.

First, step ST1 will be described. Hereinafter, a case will beillustrated where the “luminance information” is sharpness.

In the optical system of the imaging apparatus A of FIG. 18, the opticalregion D1 has a planar surface, whereas the optical region D2 has anaspherical shape. For simplicity of description, it is assumed that thelens L2 is an ideal lens free of aberration.

Since the surface of the optical region D1 is a planar surface, rayshaving passed through the optical region D1 and the lens L2 have nospherical aberration, as indicated by a solid line in FIG. 23. Whenthere is no spherical aberration, the point spread distribution varieswith an increase in shift from the focal point. In other words, thepoint spread distribution varies with changing subject distance.

Moreover, due to the aspherical shape of the optical region D2, there isspherical aberration associated with rays having passed through theoptical region D2 and the lens L2 as shown by the graph indicated by abroken line in FIG. 23. Such spherical aberration can be imparted byadjusting the aspherical shape of the optical region D2. With suchspherical aberration, in a predetermined range near the focal point ofthe lens optical system L, the point spread distribution associated withrays having passed through the optical region D2 can be keptsubstantially constant. In other words, the point spread distributioncan be kept substantially constant within the predetermined subjectdistance range.

Sharpness also changes with changes in point spread distribution. Sincethe image sharpness increases as the point image decreases in size, agraph indication of the relationship between subject distance andsharpness will result in a relationship as shown in FIG. 24. In thegraph of FIG. 24, G1 represents the sharpness in a predetermined regionof an image obtained at the pixel P1 (first image I1), and G2 representsthe sharpness in a predetermined region of an image obtained at thepixel P2 (second image I2).

Sharpness can be determined based on differences between the luminancevalues of adjacent pixels in an image block of a predetermined size.Alternatively, it may be determined based on a frequency spectrumobtained by applying Fourier transform to the luminance distribution ofan image block of a predetermined size.

When determining a sharpness E in a block of a predetermined size basedon differences between the luminance values of adjacent pixels, (math.12) is used, for example.

$\begin{matrix}{E = {\sum\limits_{i}{\sum\limits_{j}\;\sqrt{\left( {\Delta\; x_{i,j}} \right)^{2} + \left( {k\;\Delta\; y_{i,j}} \right)^{2}}}}} & \left\lbrack {{math}.\mspace{14mu} 12} \right\rbrack\end{matrix}$

In (math. 12), Δx_(i,j) is a difference value between the luminancevalue of a pixel at coordinates (i,j) and the luminance value of a pixelat coordinates (i+1,j) within an image block of a predetermined size;Δy_(i,j) is a difference value between the luminance value of a pixel atcoordinates (i,j) and the luminance value of a pixel at coordinates(i,j+2) within the image block of the predetermined size; and k is acoefficient. The reason why the luminance value of Δy_(i,j) in the ydirection is calculated by using coordinate j and coordinate j+2 isthat, in an image obtained at each of the pixels P1 and P2, luminanceinformation along the vertical direction (y direction) is created forevery other pixel. It is desirable that Δy_(i,j) is multiplied by apredetermined coefficient (e.g., k=0.5).

In each of the first and second images I1 and I2, luminance informationof the image in the y direction is missing for every other pixel. Theluminance information of any missing pixel may be interpolated from theluminance information of an adjacent pixel along the y direction. Forexample, if the luminance information at coordinates (i,j+1) is missingfrom the image, coordinates (i,j+1) may be interpolated by taking anaverage of the luminance information of coordinates (i,j) andcoordinates (i,j+2). When determining the sharpness E of coordinates(i,j+1) with (math. 1), it may be assumed that k=1, and Δy_(i,j) will bea difference value between the luminance value of a pixel at coordinates(i,j) and the luminance value of a pixel at coordinates (i,j+1) (a valueinterpolated from the luminance information of coordinates (i,j+2))within an image block of a predetermined size. From the calculation of(math. 2), the greater the difference between luminance values in theimage block of the predetermined size is, the greater sharpness isobtained.

Although image sharpness can be determined by using (math. 12) mentionedabove, it may also be determined based on a frequency spectrum obtainedby applying Fourier transform to the sharpness within the block of thepredetermined size.

FIGS. 25( a) to (c) each show a luminance distribution in an image blocksized 16×16. Sharpness decreases in the order of FIGS. 25( a), (b), (c).FIGS. 25( d) to (f) show frequency spectrums obtained by applying atwo-dimensional Fourier transform to the respective image blocks shownin FIGS. 25( a) to (c). In FIGS. 25( d) to (f), for ease ofunderstanding, the intensity of each frequency spectrum is indicatedthrough logarithmic transformation, such that the more intense thefrequency spectrum is, the brighter it appears. In each frequencyspectrum, the place of highest luminance in the center is a DCcomponent, and increasingly higher frequencies exist closer to theperipheral portion.

In FIGS. 25( d) to (f), it can be seen that the higher frequencyspectrum is lost as the image sharpness decreases. Therefore, in orderto determine sharpness from any such frequency spectrum, the entirefrequency spectrum or a portion thereof may be extracted, for example.

FIGS. 26A to 26E is a diagram describing a method of enhancing thesharpness of G2 based on the sharpness of G1. FIG. 26A shows a subject,which is a white-black chart, and FIG. 26B is a diagram showing a crosssection in the luminance of the subject of FIG. 26A. As shown in FIG.26B, the luminance of the chart has a step-like cross section; however,the image will have a luminance cross section as shown in FIG. 26C whentaken by placing the chart at a predetermined position that is shiftedslightly frontward from the subject position at which the rays reachingthe pixel P1 are best focused, for example. In the graph of FIG. 26C, G1is a luminance cross section of an image which is generated at the pixelP1, whereas G2 is a luminance cross section of an image which isgenerated at the pixel P2. Thus, the luminance cross section of G1 iscloser to the luminance cross section of the actual chart in FIG. 26Bthan is the luminance cross section of G2, therefore having a highersharpness.

When the luminance cross section of G1 with a high sharpness issubjected to a second-order differentiation, the distribution of FIG.26D is obtained, and the edge of the G1 image can be detected. Next, bysubtracting the distribution of FIG. 26D from the G2 luminancedistribution of FIG. 26C, the distribution of FIG. 26E is obtained,whereby the G2 luminance distribution has been sharpened. Now, whensubtracting the distribution of FIG. 26D, the distribution of FIG. 26Dmay be multiplied by a predetermined coefficient, which then may besubtracted from the G2 luminance distribution of FIG. 26C, thuscontrolling the degree of sharpening G2.

Although the present embodiment illustrates the image sharpening inone-dimensional terms for simplicity of description, an image istwo-dimensional and therefore a two-dimensional sharpening process isactually to take place.

Through the above image processing, the sharpness of G2 which isindicated by a solid line in FIG. 24 can be sharpened as in G2′ which isindicated by a broken line, thus sharpening the resultant color image.

FIG. 27 is a graph showing the relationship between subject distance andsharpness in the case where the surface in the optical region D2 ischanged from an aspherical shape to a spherical shape in FIG. 18. Inthis case, too, the image can be sharpened similarly to FIG. 24.

In the present embodiment, as shown in FIG. 27, different componentshave a high sharpness depending on the subject distance. Therefore,respective sharpnesses are detected from the luminance cross sections ofG1 and G2, and the component with the higher sharpness is selected tosharpen any other component.

Through the above image processing, the sharpnesses of G1 and G2 whichare indicated by solid lines in FIG. 27 can be respectively sharpened asin G1′ and G2′ which are indicated by broken lines, thus sharpening theresultant color image.

Next, another image sharpening technique will be described. FIG. 28 is adiagram describing a method of enhancing the sharpness of G1 based onG2′, which is a sharpness-enhanced version of G2. The construction ofthe optical regions D1 and D2 is the same as that in FIG. 24, and thepoint spread distribution created by rays having passed through theoptical region D2 is kept substantially constant within a predeterminedsubject distance range. Therefore, the point spread distribution whichis created by extracting the pixel P2 (G2 component) is substantiallyconstant within a predetermined subject distance range. So long as thepoint spread distribution is substantially constant in the predeterminedsubject distance range, an image which is formed by extracting the pixelP2 (G2 component) is restorable based on a predetermined point spreaddistribution, regardless of the subject distance.

Hereinafter, a method of restoring a captured image based on a pointspread distribution will be described. Assuming that the original imageis f(x,y), and the point spread distribution is h(x,y), the capturedimage g(x,y) is expressed by (math. 13).g(x,y)=f(x,y)

h(x,y) (where

represents convolution)  [math. 13]

A Fourier transform applied to both sides of (math. 13) gives (math. 3).G(u,v)=F(u,v)H(u,v)  [math. 3]

Now, by applying an inverse filter Hinv(u,v) of (math. 14) to thedeteriorated image G(u,v), a two-dimensional Fourier transform F(u,v) ofthe original image is obtained as in (math. 15). By applying an inverseFourier transform to this, the original image f(x,y) can be obtained asa restored image.

$\begin{matrix}{{{Hinv}\left( {u,v} \right)} = \frac{1}{H\left( {u,v} \right)}} & \left\lbrack {{math}.\mspace{14mu} 14} \right\rbrack \\{{F\left( {u,v} \right)} = {{{Hinv}\left( {u,v} \right)}{G\left( {u,v} \right)}}} & \left\lbrack {{math}.\mspace{14mu} 15} \right\rbrack\end{matrix}$

However, if H(u,v) is 0 or has a very small value, Hinv(u,v) willdiverge; therefore, a Wiener filter Hw(u,v) as indicated by (math. 16)is used for restoration from the deteriorated image.

$\begin{matrix}{{{Hw}\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2}}{{{H\left( {u,v} \right)}}^{2} + {{{N\left( {u,v} \right)}}^{2}/{{F\left( {u,v} \right)}}^{2}}}}} & \left\lbrack {{math}.\mspace{14mu} 16} \right\rbrack\end{matrix}$

In (math. 16), N(u,v) is noise. Since usually the noise and the originalimage F(u,v) are unknown, a constant k is actually used to restore thedeteriorated image with a filter of (math. 17).

$\begin{matrix}{{{Hw}\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2\;}}{{{H\left( {u,v} \right)}}^{2} + k}}} & \left\lbrack {{math}.\mspace{14mu} 17} \right\rbrack\end{matrix}$

With such a restoration filter, the sharpness of G2 which is indicatedby a solid line in FIG. 28 can be sharpened as in G2′ which is indicatedby a dotted line. Furthermore, in a manner similar to the method shownin FIG. 26, the G2′ luminance cross section may be subjected tosecond-order differentiation, and this may be subtracted from G1,whereby the sharpness of G1 is enhanced to result in the sharpened G1′which is indicated by a broken line in FIG. 28.

Through the above image processing, the sharpness of G2 and thesharpness of G1 which are indicated by solid lines in FIG. 28 can berespectively sharpened as in G2′ indicated by a dotted line and G1′indicated by a broken line. Through such a sharpening process, the depthof field can be expanded from the sharpening process described in FIG.24.

Next, the depth map generation at step ST2 in FIG. 22 will bespecifically described. The depth map is generated by determining asubject distance for each predetermined region (each calculation block)in a captured image.

To the second signal processing section C2 shown in FIG. 18, the firstimage I1 (shown in FIG. 18) obtained by extracting the pixel P1 (G1component) and the second image obtained by extracting the pixel P2 (G2component) are input. Since the two optical regions D1 and D2 havemutually different optical characteristics, the image sharpnesses(values calculated by using luminance) of the first and second images I1and I2 vary depending on the subject distance. In the storage section Me(shown in FIG. 18), a correlation between sharpness and subject distanceof light having passed through each of the optical regions D1 and D2 isstored. In the third signal processing section C3 (shown in FIG. 18),the distance from the subject can be determined based on the sharpnessesof the first and second images I1 and I2 and the aforementionedcorrelations.

Herein, the range Z in FIG. 24 and FIG. 28 represents a region in whichG1 changes but in which G2 hardly changes. In the range Z, the subjectdistance can be determined by utilizing this relationship. For example,in the range Z, since there is a correlation between the subjectdistance and the ratio between the sharpnesses G1 and G2, thecorrelation between the subject distance and the ratio between thesharpnesses G1 and G2 is stored in the storage section Me in advance.

When the imaging apparatus is used, within the data (captured image)which is obtained through a single instance of imaging, a ratio insharpness between the first image I1 which is generated from the pixelP1 (G1 component) alone and the second image I2 which is generated fromthe pixel P2 (G2 component) alone is determined for each calculationblock. Then, by using the correlations stored in the storage section Me,the subject distance can be determined. Specifically, for eachcalculation block, the sharpness ratio in the aforementioned correlationand the sharpness ratio values of the first image I1 and the secondimage I2. Then, a subject distance that corresponds to a matching valuebetween the two is regarded as the distance from the subject atshooting.

In order to univocally determine the subject distance from the ratiobetween the sharpness of the first image I1 generated from the pixel P1alone and the sharpness of the second image I2 generated from the pixelP2 alone, it is necessary that the sharpness ratio always variesthroughout a predetermined subject distance range.

In FIG. 24, FIG. 27, and FIG. 28, the sharpness ratio always variesthroughout the range Z, and thus the subject distance can be univocallydetermined. Moreover, since the sharpness values being too low wouldmake it impossible to determine a ratio, the sharpness values may beequal to or greater than a certain value.

Note that the relationship between subject distance and sharpness isdetermined from the radii of curvature, aspheric coefficients, andrefractive indices of the optical regions D1 and D2. In other words, theoptical regions D1 and D2 need to have optical characteristics such thatthe ratio between the sharpness of the first image I1 and the sharpnessof the second image I2 always varies throughout the predetermineddistance range.

In the present embodiment, the subject distance may be determined byusing any value other than sharpness, e.g., contrast, so long as it is avalue that is calculated by using luminance (luminance information).Contrast can be determined from a ratio between the highest luminancevalue and the lowest luminance value within a predetermined calculationblock, for example. Sharpness is a difference between luminance values,whereas contrast is a ratio between luminance values. A contrast may bedetermined from a ratio between a point of the highest luminance valueand a point of the lowest luminance value, or a contrast may bedetermined from a ratio between an average value of several points ofthe largest luminance values and an average value of several points ofthe lowest luminance values. In the case of using contrast to determinethe subject distance, similarly to the case of sharpness, a correlationbetween the subject distance and the contrast ratio is stored in advancein the storage section Me. By determining a contrast ratio between thefirst image I1 and the second image I2 for each calculation block, it ispossible to determine the subject distance by utilizing the correlation.

Moreover, in the present embodiment, the subject distance may bedetermined by using a value other than sharpness or contrast, e.g.,point spread distribution. Hereinafter, a method of determining a pointspread distribution from the first image I1 and the second image I2 willbe described.

When the aforementioned (math. 17) is used to restore the second imageI2 generated from the pixel P2 (G2 component) alone, a restored imagei2′(x,y) which is very close to the original image f(x,y) is obtained.Now, assuming a first image i1(x,y) which is generated from the pixel P1(G1 component) alone, and a point spread distribution h1(x,y) associatedwith rays passing through the region D1, it is possible to expressI1(x,y) by (math. 18).i1(x,y)≈i 2′(x,y)

h1(x,y) (where

represents convolution)  [math. 18]

A Fourier transform applied to both sides of (math. 18) gives (math.19).I1(u,v)≈I2′(u,v)H1(u,v)  [math. 19]

Through transformation of (math. 19), frequency domain values H1(u,v) ofthe point spread distribution h1(x,y) are obtained as in (math. 20).

$\begin{matrix}{{H\; 1\left( {u,v} \right)} \approx \frac{I\; 1\left( {u,v} \right)}{I\; 2^{\prime}\left( {u,v} \right)}} & \left\lbrack {{math}.\mspace{14mu} 20} \right\rbrack\end{matrix}$

By applying an inverse Fourier transform to this, a point spreaddistribution h1(x,y) associated with rays passing through the region D1can be obtained.

Since the point spread distribution h1(x,y) associated with rays passingthrough the region D1 varies with subject distance, the point spreaddistribution h1(x,y) and the subject distance have a correlation. Thiscorrelation can be utilized to determine the subject distance.

In the case of using a representative value to express a point spreaddistribution, the diameter of the point spread distribution can be used,for example. Similarly to the case of sharpness or contrast, acorrelation between subject distance and point image diameter is storedin advance in the storage section Me. By determining a point spreaddistribution from the first image I1 or the second image I2 for eachblock, and determining the diameter of the point image from the pointspread distribution, the subject distance can be determined throughcorrelation. The diameter of a point image can be determined from thehalf-width of the point spread distribution, for example.

The present embodiment may be constructed so as to generate an imageobtained through addition of the first image I1 and the second image I2in the case where the radii of curvature of the respective regions aremade different from each other as shown in FIG. 27. The distance rangein which sharpness attains a certain value or greater in the imagegenerated through addition of the first image I1 and the second image I2is larger than those of the first image and the second image. In thiscase, the ratio between the sharpness of the image generated throughaddition and the sharpness of either the first image I1 or the secondimage I2 has a correlation with subject distance. By storing thiscorrelation in advance, it is possible to determine a subject distancefor each predetermined region of an image.

By calculating a subject distance(s) in a captured image, and expressingthe subject distance(s) in a monochromatic luminance value(s) (e.g., 256gray scale levels), an image representing the depth information isobtained; this is the depth map. FIG. 29( a) is a subject image(captured image) according to the present embodiment, whereas FIG. 29(b) is a depth map of the subject image of FIG. 29( a). This is arepresentation in 256 gray scale levels, where the whiter the morefrontward, and the blacker the more rearward the subject exists. In FIG.29( b), the completely black portions in the check pattern are places oferror in distance measurement. In a subject image, in any place with abroadly uniform luminance value, no change in sharpness occurs near itscenter, thereby making distance measurement impossible. However,refocused image generation is not affected by any place where distancemeasurement is impossible because, near the center of a range having abroadly uniform luminance value, the image sharpness would not changeirrespective of whether a refocus calculation is applied or not. It isnot necessary that the depth map is in 256 gray scale levels; it may bea 16-bit (i.e. 65536 gray scale levels) image; it does not even need tobe image data, but may be numerical data based on distance. It may alsocontain negative values so long as relative positioning of subjects canbe indicated.

Next, the PSF generation at step ST3 in FIG. 22 will be specificallydescribed. PSF is generated for each subject position (subjectdistance), for example. Furthermore, a PSF may be generated for eachangle of view (pixel or predetermined region).

It may do well to express the PSF shape by a mathematical expression,e.g., a Gaussian distribution (Gaussian function) as indicated by (math.21), because it is possible to obtain a PSF at any arbitrary subjectposition on the fly, through simple calculation according to themathematical expression; this way, there is no need to store enormoussubject PSF data in a memory in advance.

$\begin{matrix}{{{{Weight}\left( {i,j} \right)} = {\exp\left( {- \frac{i^{2} + j^{2}}{2\;\sigma^{2}}} \right)}}{\sigma = {{k \cdot d} + 0.001}}} & \left\lbrack {{math}.\mspace{14mu} 21} \right\rbrack\end{matrix}$Herein, i is a lateral coordinate in the PSF; j is a vertical coordinatein the PSF; and (i,j)=(0,0) represents the center of the PSF.Weight(i,j) is the intensity (weight) of the PSF at i,j; and d is thesubject distance, such that the best focus position is expressed as theorigin (d=0). As used herein, the “best focus position” refers to asubject position (subject distance) at which intensity change in the PSFtakes a local maximum. If the “intensity change in the PSF” is large,the PSF has a sharp peak, e.g., the peak has a small half-width. If the“intensity change in the PSF” is small, the PSF has a gentle peak; e.g.,the peak may have a large half-width. Moreover, k is a coefficient forgain adjustment, which adjusts the intensity change in the PSF. Theaddition “0.001” to σ is a constant for preventing divergence when(i,j)=(0,0), which has been selected to be a sufficiently small valuerelative to k·d. This constant does not need to be “0.001”, and may bechanged as appropriate.

FIG. 30 is a PSF cross-sectional intensity distribution as determined bythe Gaussian distribution of (math. 21), where the plotting is madebased on j=0, i=−5 to 5, σ=1.4. When i=0, j=0, the PSF has the highestintensity and has a distribution which is symmetric between right andleft (rotation symmetric). Although it is not necessary that theintensity distribution of the PSF is rotation symmetric, rotationalsymmetry would be desirable in order to generate a non-biased naturalrefocused image.

Intensity change (how sharp it is) in the PSF is adjusted by k. It mustbe set so that the PSF is sharpest when the subject position is at thebest focus position and becomes more gentle as the subject positionbecomes farther away from the best focus position. The particularsubject position to become the best focus position can be arbitrarilyset. The best focus position may be externally input by a user, ordetermined by the second signal processing section C2. In the case wherethe user determines the best focus position, the user may select aregion in the image, and the second signal processing section C2 maydetermine the subject distance in the region that has been selected bythe user and designate it as the best focus position. Alternatively, theuser may directly choose the subject distance. Once the best focusposition is determined, that subject position is defined as the origin.

FIG. 31 is a conceptual diagram of changes in the PSF cross-sectionalintensity distribution when focusing on the subject position d2. In FIG.31A, the gradient of PSF intensity is gentler at a subject position d3,which is slightly distant from the subject position d2, than at thesubject position d2. The gradient of PSF intensity is even gentler at amore distant subject position d1. By setting the origin of d (=0) at thebest focus position in (math. 21), it is ensured that the absolute valueof a increases as the subject position becomes more distant from thebest focus position, thus allowing a more gentle gradient of PSFintensity to be set. Moreover, by increasing the k value in (math. 21),it is possible to adjust the degree of change in the intensitydistribution of the PSF relative to subject position. FIG. 31Billustrates a case where the k value is increased from FIG. 31A. In FIG.31B, the intensity distribution of the PSF changes more drastically withthe subject position than in FIG. 31A; and, given the same subjectposition d1 (or d3), the intensity distribution of the PSF has a gentlergradient in FIG. 31B. The k value may be thus adjusted as appropriate,and changes in the blur of an image as described later may be adjusted.Although the σ equation in (math. 21) undergoes linear change against d,a non-linear function such as a quadric function or a polynomial mayalso be used, other than a linear function. Using a non-linear functionmakes it possible to non-linearly adjust intensity changes in the PSFagainst the subject position d, i.e., changes in blur.

Although FIGS. 31A and 31B shows one cross section of PSF intensity, aPSF is two-dimensional data also having expanse in the depth direction.For the actual calculation, a two-dimensional matrix of intensity asshown in FIG. 32 may well be used. With (i,j)=(0,0) being the origin,(math. 21) can be used for the calculation. It is desirable that thenumber of rows and the number of columns in the matrix are the same,both of which are desirably odd numbers, because this will allow oneorigin to be set in the center of the matrix, and provide a PSF which isrotation symmetric around that axis. Although the number of rows and thenumber of columns in the matrix may be arbitrary, the greater they are,the greater the amount of blur can be. On the other hand, as the numberof rows and the number of columns in the matrix decrease, thecalculation time can be reduced. In FIG. 32, (a) shows a 3×3 matrix, (b)shows a 5×5 matrix, and (c) shows a 7×7 matrix of two-dimensionaldistribution of PSF intensity. Similarly to FIG. 30, it is assumed thatσ=1.4. The values in the matrix may well be normalized so that thematrix product equals 1. Specifically, after calculating valuesaccording to (math. 21), a product of all components in the matrix maybe calculated, and each component may be divided by that product. Thereason for performing normalization is to prevent change in luminance ofthe image after convolution in the subsequent refocus. By normalizingthe intensity product of the PSF to 1, it becomes possible to maintain aconstant image brightness in both the image before the refocusing andthe image after the refocusing. Normalization of PSF data may beperformed at the time of PSF calculation, or performed immediatelybefore the refocusing process. For reference, FIG. 33A shows an imageexpression in 256 gray scale levels of numerical data of the PSFintensity distribution of the 7×7 matrix in FIG. 32( c). Likewise, FIG.33B is a three-dimensional graph of FIG. 32( c).

Instead of using a mathematical expression, the actual PSF valuepertaining to the optical system might be used for the PSF calculation;in this case, however, the need to previously calculate a PSF for eachsubject distance through simulation at every certain interval requiresan enormous amount of memory for the database. On the other hand, byusing a Gaussian distribution in the form of a mathematical expression,it becomes possible to generate a PSF at any arbitrary subject positionat the time of refocus calculation, thus saving memory and reducingcalculation time. Moreover, when expressed as a Gaussian distribution,the PSF when the subject position is at the best focus position willhave 1 at the center and be surrounded by 0's, which means that theimage at the best focus position will not be deteriorated. In otherwords, intensity change in the PSF at the best focus position is greaterthan the intensity change in the PSF at any other subject position, andintensity change in the PSF becomes smaller as the subject positionbecomes away from the best focus position along the subject distancedirection.

Note that the mathematical expression representing a PSF may be anequation other than that of a Gaussian distribution. For example, it maybe an equation of a non-spherical surface that contains higher orders.

Next, the refocused image generation at step ST4 of FIG. 22 will bedescribed. This process is performed, using the subject distancesdetermined at step ST2 and the PSFs generated at step ST3, by the thirdsignal processing section C3. In correspondence with the depth map ofFIG. 29( b), a PSF convolution process is performed for each pixel ofthe sharpened image. For example, if the subject position at a givenpixel (i,j)=(i₀,j₀) in the depth map is d1 in FIG. 31, a convolutioncalculation is performed by using the PSF at d1 for a matrix which iscentered around the pixel (i₀,j₀) of the sharpened image (=a matrixhaving the same numbers of rows and columns as in the PSF). Thisoperation is performed for every pixel of the sharpened image. Thisprocess provides a refocused image with internal variety such that focusis placed only on a desired place(s) while leaving anything elseblurred, as opposed to the sharpened image, which has little image blurat any and all subject positions.

FIG. 34 is an image obtained by refocusing the subject image of FIG. 29(a) based on the depth map of FIG. 29( b). Processing was performed byassuming that: the best focus position (corresponding to d2 in FIG. 31)is the oranges in the front; the PSF matrix has 15×15 regions; k=1. Itcan be seen that the image of FIG. 29( a) is uniformly focused, whereasthe refocused image of FIG. 34 has a frontward focus, the remainingbackground becoming more blurred as it gets farther away. Note that, inFIG. 29( b), any place of distance measurement detection error may beexempted from the refocusing process through an exception handling.Alternatively, they may be subjected to refocusing by using a value atan arbitrary subject distance because, since those are regions with aconstant luminance value, sharpness will not change irrespective ofwhether refocusing is applied or not.

In the present embodiment, within step ST1, only the process ofdetermining image sharpness (luminance information) may be performed,while omitting the image sharpening process. In this case, the image(captured image) which has been acquired from the sensors (photodiodes)may directly be subjected to a refocusing process. Herein, the imageacquired from the sensors may be the first or second image I1 or I2shown in FIG. 18, or an image that contains images from the first andsecond pixels P1 and P2. In the case of omitting the sharpening process,it is preferable to use the G1 image (first image I1) having the highersharpness in FIG. 28. Such a process is especially effective in the casewhere blurred portions are supposed to become even more blurred forfurther emphasis.

Moreover, only specific regions of the image may be subjected to arefocusing process. The calculation time can be reduced by processingonly the portions where blur is desired.

Without necessarily using a PSF, for example, regions where blur isdesired may be exclusively subjected to a spatial filter process, e.g.,an averaging filter, thus creating blur. Moreover, region wheresharpening is desired may be exclusively subjected to a spatial filterprocess, e.g., a sharpening filter, thus sharpening the subject image ofinterest. In these cases, without performing step ST3 of the flowchartshown in FIG. 22, regions where blur is desired (or regions wheresharpening is desired) may be determined based on the depth map at stepST4, and a spatial filter process may be performed.

Now, an exemplary method of refocused image generation in the case wherethe sharpening process of step ST1 in FIG. 22 is omitted will bedescribed.

At step ST1, after obtaining luminance information of the image, apredetermined region with the highest sharpness (i.e., focused) isdetected. Then, based on the depth map generated at step ST2, a blurringprocess is performed for every predetermined region, in accordance withthe distance from a subject that has been detected as the region withthe highest sharpness. For example, a blurring process may be performedso that more blur is applied to regions which are located at farther(rather than closer) distance from the subject that has been detected asthe region with the highest sharpness. As a result, places which are notfocused and thus are blurred can be made more blurred for emphasis.Moreover, the region which has been detected as the region with thehighest sharpness may be sharpened by using a restoration filter or aspatial filter. As a result, the sharp region and the blurred regionswithin a captured image can be more emphasized. Note that, whensharpening is performed by using a restoration filter in this method,the PSF (point spread distribution) used may be retained in the form ofa mathematical function, or what is determined in advance for eachsubject distance from the characteristics of the optical system may beretained for use. More desirably, PSFs for different angles of view maybe retained for use, in order to realize sharpening with a higherprecision.

Moreover, convolution at the end portions of an image may be separatelyhandled by a branched calculation process because of there being scarcepixels in the original image. For example, a part of a PSF may be usedso as to be adapted to a partial vignetting at the image end portions.

As the PSF convolution calculation process, Fourier transform may beused. For example, DFT (Discrete Fourier Transform) or FFT (Fast FourierTransform) may be used, whereby the calculation time can be reduced.This is especially effective when there is a broad region (predeterminedregion) in which the subject distance remains constant, where the regionwith the constant subject distance is to be regarded as one block in thecalculation. For example, a matrix of PSFs matching the block size ofthe image for calculation may be generated, and each may be subjected toa Fourier transform so that a calculation may be performed in thefrequency space. Once subjected to a Fourier transform, a convolutioncalculation requires much less calculation because, in the frequencyspace, calculation can be achieved via multiplication between respectivecomponents. After obtaining a multiplication product in the frequencyspace, it may be subjected to an inverse Fourier transform, whereby animage similar to what would be obtained through a convolutioncalculation can be obtained.

Note that the optical system of the imaging apparatus of the presentembodiment may be an image-side telecentric optical system. As a result,even if the angle of view changes, incidence occurs with theprincipal-ray incident angle of the array optical device K having avalue close to 0 degrees, so that the crosstalk between light beamsreaching the pixels P1 and P2 can be reduced across the entire imagingregion.

Although the present embodiment has illustrated the lens L2 to be anideal lens for simplicity of description as mentioned above, it is notnecessary to employ an ideal lens.

Although the optical element L1 and the lens L2 are separate in thepresent embodiment, another possible construction is where the lens L2has the optical regions D1 and D2, with the optical element L1 beingeliminated. In this case, the stop S may be disposed near the opticalregions D1 and D2 of the lens L2.

Thus, according to the present embodiment, through (e.g. a singleinstance of) imaging using a single imaging system, both an image andthe subject distance can be obtained. Since a subject distance can becalculated for each calculation block, it is possible to acquire thesubject distance at any arbitrary position in the image. Therefore, itis also possible to acquire a depth map across the entire image. Thus,after the capturing has been done, it is possible to focus on everysubject in the image.

Moreover, the distance from the subject can be obtained with a singleimaging system, which is unlike in an imaging apparatus having aplurality of imaging optical systems, where it would be necessary toensure matching characteristics and positions between the plurality ofimaging optical systems. Moreover, when a motion video is shot by usingthe imaging apparatus of the present embodiment, an accurate distancefrom the subject can be measured even if the subject position changeswith lapse of time.

Embodiment 5

This Embodiment 5 differs from Embodiment 4 in that a plurality of bestfocus positions are provided discretely. In the present embodiment, anydetailed description directed to similar subject matter to Embodiment 4will be omitted.

In the present embodiment, as shown in FIG. 35, best focus position areset at two places or an arbitrary plural number of places. In additionto the position d2, the position d4 is also a best focus position.Although the position d5 is in between the position d2 and the positiond4, its intensity distribution of the PSF is gentler than those of thepositions d2 and d4. To provide “a plurality of best focus positionsdiscretely” means that there exist a plurality of points at whichintensity change in the PSF takes a local maximum (best focuspositions), such that intensity change in any PSF between the pluralityof best focus positions is smaller than the intensity change at the bestfocus positions. Note that the size of intensity change in the PSF maybe different between the plurality of best focuses.

In order to set best focus positions at two places, a may be expressedin a quartic function in (math. 21). It is not necessary to employ aquartic function; any higher order, or an exponential or logarithmicexpression may be used. By using the method shown in FIG. 35, in animage in which two people, i.e., one person in the close neighborhoodand one person in the distance, are captured, it becomes possible tofocus on both of the person in the close neighborhood and the person inthe distance, while blurring any other background. This is a techniquewhich cannot be achieved with conventional optical systems. For example,even with a blur effect that is attained by a single-lens reflex camerahaving a very small Fno, it is only possible to achieve focus at onesubject position in the close neighborhood, in the distance, orsomewhere in between. Without being limited to two arbitrary places,objects at any larger plural number of places may be chosen for bestfocus, while blurring anything else.

Embodiment 6

This Embodiment 6 differs from Embodiment 4 in that filters havingspectral transmittance characteristics are provided on the pixels. Inthe present embodiment, any description directed to similar subjectmatter to Embodiment 4 will be omitted.

FIG. 36 is a schematic diagram showing an imaging apparatus A accordingto Embodiment 6. The imaging apparatus A of the present embodimentincludes: a lens optical system L having an optical axis V; an arrayoptical device K disposed near the focal point of the lens opticalsystem L; an imaging device N; a second signal processing section C2; athird signal processing section C3; a first signal processing sectionC1; and a storage section Me.

FIG. 37A is an enlarged diagram showing the array optical device K andthe imaging device N shown in FIG. 36, and FIG. 37B is a diagram showingrelative positioning between the array optical device K and pixels onthe imaging device N. The array optical device K is disposed so that theface on which the optical elements M1 are formed is oriented toward theimaging plane Ni. Pixels P are disposed in a matrix shape on the imagingplane Ni. The pixels P can be classified into pixels P1, P2, P3, and P4.

Filters having first spectral transmittance characteristics are providedon the pixels P1 and P2, so as to mainly allow rays of the green band topass through, while absorbing rays in any other band. A filter havingsecond spectral transmittance characteristics is provided on the pixelP3, so as to mainly allow rays of the red band to pass through, whileabsorbing rays in any other band. A filter having third spectraltransmittance characteristics is provided on the pixel P4, so as tomainly allow rays of the blue band to pass through, while absorbing raysin any other band.

Pixels P1 and pixels P3 alternate within the same row. Moreover, pixelsP2 and pixels P4 alternate within the same row. Rows of pixels P1 and P3and rows of pixels P2 and P4 alternate along the vertical direction(column direction). Thus, the plurality of pixels P1, P2, P3, and P4compose a Bayer pattern. In the case where the pixels P1, P2, P3, and P4are arranged in a Bayer pattern, the pixel P1 and the pixel P2 bothhaving a filter transmitting light of the green band are disposed atoblique positions in the plane of the imaging plane Ni. The positions ofthe pixel P3 and the pixel P4 may be reversed.

The array optical device K is arranged so that one optical element M1thereof corresponds to two rows of pixels, consisting of one row ofpixels P1 and P3 and one row of pixels P2 and P4, on the imaging planeNi. Microlenses Ms are provided on the imaging plane Ni so as to coverthe surface of the pixels P1, P2, P3, and P4.

The array optical device K is designed so that: a large part of thelight beam B1 having passed through the optical region D1 (shown in FIG.36, FIG. 19) on the optical element L1 (the light beam B1 indicated bysolid lines in FIG. 36) reaches the pixels P1 and P3 on the imagingplane Ni; and a large part the light beam having passed through theoptical region D2 (the light beam B2 indicated by broken lines in FIG.36) reaches the pixels P2 and P4 on the imaging plane Ni. Specifically,the above construction is realized by appropriately setting parameterssuch as the refractive index of the array optical device K, the distancefrom the imaging plane Ni, and the radius of curvature of the surface ofthe optical elements M1.

The stop S is a region through which light beams of all angles of viewwill pass. Therefore, by inserting a surface having opticalcharacteristics for controlling the focusing characteristic in theneighborhood of the stop S, it becomes possible to control the focusingcharacteristic of light beams of all angles of view alike. In otherwords, in the present embodiment, the optical element L1 may be providedin the neighborhood of the stop S. By disposing the optical regions D1and D2 having optical characteristics which provide mutually differentfocusing characteristics in the neighborhood of the stop S, the lightbeam can be allowed to have a focusing characteristic that is inaccordance with the number of divided regions.

In FIG. 36, the optical element L1 is provided at a position forallowing light having passed through the optical element L1 to beincident on the stop S directly (i.e., not via any other opticalmember). The optical element L1 may be provided on the imaging device Nside of the stop S. In that case, the optical element L1 may be providedbetween the stop S and the lens L2, so that light having passed throughthe stop S is incident on the optical element L1 directly (i.e., not viaany other optical member).

Moreover, the array optical device K has a function of branching outinto outgoing directions depending on the incident angle of the ray.Therefore, the light beam can be branched out over the pixels on theimaging plane Ni so as to correspond to the optical regions D1 and D2 asdivided near the stop S.

The first signal processing section C1 (shown in FIG. 36) generates acolor image by using luminance information from the plurality of pixelsP1, P2, P3, and P4. Hereinafter, the specific method of color imagegeneration will be described.

In the optical system of the imaging apparatus A of FIG. 36, the opticalregion D1 has a planar surface, whereas the optical region D2 has anaspherical shape. For simplicity of description, it is assumed that thelens L2 is an ideal lens free of aberration.

Since the surface of the optical region D1 is a planar surface, there isno spherical aberration associated with rays having passed through theoptical region D1 and the lens L2, as indicated by a solid line in thegraph of FIG. 23. When there is no spherical aberration, the pointspread distribution varies with an increase in shift from the focalpoint. In other words, the point spread distribution varies withchanging subject distance.

Moreover, due to the aspherical shape of the optical region D2, there isspherical aberration associated with rays having passed through theoptical region D2 and the lens L2 as shown by the graph indicated by abroken line in FIG. 23. Such spherical aberration can be imparted byadjusting the aspherical shape of the optical region D2. With suchspherical aberration, in a predetermined range near the focal point ofthe lens optical system L, the point spread distribution associated withrays having passed through the optical region D2 can be keptsubstantially constant. In other words, the point spread distributioncan be kept substantially constant within the predetermined subjectdistance range.

Sharpness also changes with changes in point spread distribution. Sincethe image sharpness increases as the point image decreases in size, agraph indication of the relationship between subject distance andsharpness will result in a relationship as shown in FIG. 38. In thegraph of FIG. 38, G1 and R respectively represent the sharpnesses in apredetermined region of images generated at the pixels P1 (greencomponent) and P3 (red component), whereas G2 and B respectivelyrepresent the sharpnesses in a predetermined region of images generatedat the pixels P2 (green component) and P4 (blue component).

Sharpness can be determined based on differences between the luminancevalues of adjacent pixels in an image block of a predetermined size.Alternatively, it may be determined based on a frequency spectrumobtained by applying Fourier transform to the luminance distribution ofan image block of a predetermined size.

When determining a sharpness E in a block of a predetermined size foreach component of the pixels P1, P2, P3, and P4 based on differencesbetween the luminance values of adjacent pixels, (math. 22) is used, forexample.

$\begin{matrix}{E = {\sum\limits_{i}{\sum\limits_{j}\sqrt{\left( {\Delta\; x_{i,j}} \right)^{2} + \left( {\Delta\; y_{i,j}} \right)^{2}}}}} & \left\lbrack {{math}.\mspace{14mu} 22} \right\rbrack\end{matrix}$

Since the pixels P1, P2, P3, and P4 compose a Bayer pattern as mentionedearlier, the sharpness of each component is to be determined through acalculation by extracting pixel information from every other pixel alongboth the x direction and the y direction of the image.

In (math. 22), Δx_(i,j) is a difference value between the luminancevalue of a pixel at coordinates (i,j) within an image block of apredetermined size and the luminance value of a pixel at coordinates(i+2,j); and Δy_(i,j) is a difference value between the luminance valueof a pixel at coordinates (i,j) and the luminance value of a pixel atcoordinates (i,j+2), within the image block of the predetermined size.

From the calculation of (math. 22), the greater the difference betweenluminance values in the image block of the predetermined size is, thegreater sharpness is obtained.

When generating a color image, the color image may be generated bysimply interpolating the chromatic information that is lost for eachpixel position on the basis of the luminance information of the pixelsP1, P2, P3, and P4; however, the sharpnesses of G2 and B is smaller thanthe sharpnesses of G1 and R as shown in FIG. 38, and therefore the colorimage may be generated after enhancing the sharpnesses of G1 and R.

FIGS. 39A to 39B is a diagram describing a method of enhancing thesharpnesses of G2 and B based on the sharpnesses of G1 and R. FIG. 39Ashows a subject, which is a white-black chart, and FIG. 39B is a diagramshowing a cross section in the luminance of the subject of FIG. 39A. Asshown in FIG. 39B, the luminance of the chart has a step-like crosssection; however, the image will have a luminance cross section as shownin FIG. 39C when taken by placing the chart at a predetermined positionthat is shifted slightly frontward from the subject position at whichthe rays reaching the pixels P1 and P3 are best focused, for example. Inthe graph of FIG. 39C, G1 and R are luminance cross sections of imagesgenerated at the pixels P1 (green component) and P3 (red component),respectively, whereas G2 and B are luminance cross sections of imagesgenerated at the pixels P2 (green component) and P4 (blue component),respectively. Thus, the luminance cross sections of G1 and R are closerto the luminance cross section of the actual chart in FIG. 39B than arethe luminance cross sections of G2 and B, therefore having a highersharpness.

When a white-black chart such as that shown in FIG. 39A is imaged, theluminance cross section of G1 and the luminance cross section of R willhave substantially identical cross sections; in actuality, however, asubject image of every possible color component will be taken, and theluminance cross sections of G1 and R in FIG. 39C will not coincide inmost cases. Therefore, the respective sharpnesses may be detected fromthe luminance cross sections of G1 and R, and a color component with ahigh sharpness may be selected to sharpen the luminance cross sectionsof G2 and B. When a luminance cross section with a high sharpness isselected, and its luminance cross section is subjected to second-orderdifferentiation, the distribution of FIG. 39D is obtained, and the edgeof an image of the color component with a high sharpness can bedetected. Next, by subtracting the distribution of FIG. 39D from therespective G2 and B luminance distributions of FIG. 39C, thedistribution of FIG. 39E, whereby the G2 and B luminance distributionshave been sharpened. Now, when subtracting the distribution of FIG. 39D,the distribution of FIG. 39D may be multiplied by a predeterminedcoefficient, which then may be subtracted from the G2 and B luminancedistributions of FIG. 39C, thus controlling the degree of sharpening G2and B.

Although the present embodiment illustrates the image sharpening inone-dimensional terms for simplicity of description, an image istwo-dimensional and therefore a two-dimensional sharpening process isactually to take place.

Through the above image processing, the sharpnesses of G2 and B whichare indicated by a solid line in FIG. 38 can be sharpened as in G2′ andB′ which is indicated by a broken line, thus sharpening the resultantcolor image.

FIG. 40 is a graph showing the relationship between subject distance andsharpness in the case where the surface in the optical region D2 ischanged from an aspherical shape to a spherical shape in FIG. 36. Inthis case, too, the color image can be sharpened similarly to FIG. 38.

In the present embodiment, as shown in FIG. 40, different colorcomponents have a high sharpness depending on the subject distance.Therefore, respective sharpnesses are detected from the luminance crosssections of G1, G2, R, and B, and the color component with the highestsharpness is selected to sharpen any other color component.

Through the above image processing, the sharpnesses of G1, G2, R, and Bwhich are indicated by solid lines in FIG. 40 can be respectivelysharpened as in G1′, G2′, R′, and B′ which are indicated by brokenlines, thus sharpening the resultant color image.

Next, another image sharpening technique will be described. FIG. 41 is adiagram describing a method of enhancing the sharpnesses of G1 and Rbased on G2′ and B′, which are sharpness-enhanced versions of G2 and B.The construction of the optical regions D1 and D2 is the same as that inFIG. 38, and the point spread distribution created by rays having passedthrough the optical region D2 is kept substantially constant within apredetermined subject distance range. Therefore, the point spreaddistribution which is created by extracting the respective pixels P2 (G2component) and P4 (B component) is substantially constant within apredetermined subject distance range. So long as the point spreaddistribution is substantially constant in the predetermined subjectdistance range, an image which is formed by extracting the pixels P2 (G2component) and P4 (B component) is restorable based on a predeterminedpoint spread distribution, regardless of the subject distance.

Hereinafter, a method of restoring a captured image based on a pointspread distribution will be described. Assuming that the original imageis f(x,y), and the point spread distribution is h(x,y), the capturedimage g(x,y) is expressed by (math. 23).g(x,y)=f(x,y)

h(x,y) (where

represents convolution)  [math. 23]

A Fourier transform applied to both sides of (math. 23) gives (math.24).G(u,v)=F(u,v)H(u,v)  [math. 24]

Now, by applying an inverse filter Hinv(u,v) of (math. 25) to thedeteriorated image G(u,v), a two-dimensional Fourier transform F(u,v) ofthe original image is obtained as in (math. 26). By applying an inverseFourier transform to this, the original image f(x,y) can be obtained asa restored image.

$\begin{matrix}{{{Hinv}\left( {u,v} \right)} = \frac{1}{H\left( {u,v} \right)}} & \left\lbrack {{math}.\mspace{14mu} 25} \right\rbrack \\{{F\left( {u,v} \right)} = {{{Hinv}\left( {u,v} \right)}{G\left( {u,v} \right)}}} & \left\lbrack {{math}.\mspace{14mu} 26} \right\rbrack\end{matrix}$

However, if H(u,v) is 0 or has a very small value, Hinv(u,v) willdiverge; therefore, a Wiener filter Hw(u,v) as indicated by (math. 27)is used for restoration from the deteriorated image.

$\begin{matrix}{{{Hw}\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2}}{{{H\left( {u,v} \right)}}^{2} + {{{N\left( {u,v} \right)}}^{2}/{{F\left( {u,v} \right)}}^{2}}}}} & \left\lbrack {{math}.\mspace{14mu} 27} \right\rbrack\end{matrix}$

In (math. 27), N(u,v) is noise. Since usually the noise and the originalimage F(u,v) are unknown, a constant k is actually used to restore thedeteriorated image with a filter of (math. 28).

$\begin{matrix}{{{Hw}\left( {u,v} \right)} = {\frac{1}{H\left( {u,v} \right)}\frac{{{H\left( {u,v} \right)}}^{2}}{{{H\left( {u,v} \right)}}^{2} + k}}} & \left\lbrack {{math}.\mspace{14mu} 28} \right\rbrack\end{matrix}$

With such a restoration filter, the sharpnesses of G2 and B which areindicated by a solid line in FIG. 41 can be sharpened as in G2′ and B′which is indicated by a dotted line. Furthermore, in a manner similar tothe method shown in FIG. 39, respective sharpnesses may be detected fromthe G2′ and B′ luminance cross sections; the luminance cross section ofa color component with a high sharpness may be subjected to second-orderdifferentiation; and this may be subtracted from G1 and R, whereby thesharpnesses of G1 and R are enhanced to result in the sharpened G1′ andR′ which are indicated by a broken line in FIG. 41.

Through the above image processing, the sharpnesses of G2 and B and thesharpnesses of G1 and R which are indicated by solid lines in FIG. 41can be sharpened as in G2′ and B′ which is indicated by a dotted lineand as in G1′ and R′ which is indicated by a broken line, thussharpening the resultant color image. Through such a sharpening process,the depth of field can be expanded from the sharpening process describedin FIG. 38.

Note that the optical system of the imaging apparatus of the presentembodiment may be an image-side telecentric optical system. As a result,even if the angle of view changes, incidence occurs with theprincipal-ray incident angle of the array optical device K having avalue close to 0 degrees, so that the crosstalk between light beamsreaching the pixels P1, P2, P3, and P4 can be reduced across the entireimaging region.

Although the present embodiment has illustrated the lens L2 to be anideal lens for simplicity of description as mentioned above, it is notnecessary to employ an ideal lens. For example, a non-ideal lens wouldhave axial chromatic aberration, but it is possible to select a colorcomponent with a high sharpness to sharpen other color components asdescribed earlier; thus, a color image with sharpness can be generatedeven without an ideal lens. Moreover, in the case of determining thesubject distance, the distance is to be determined based on a singlecolor component (which in the present embodiment is the greencomponent); thus, there may be some axial chromatic aberration.

Although the optical element L1 and the lens L2 are separate in thepresent embodiment, another possible construction is where the lens L2has the optical regions D1 and D2, with the optical element L1 beingeliminated. In this case, the stop S may be disposed near the opticalregions D1 and D2 of the lens L2.

Thus, according to the present embodiment, through (e.g. a singleinstance of) imaging using a single imaging system, both a color imageand the subject distance can be obtained. Since the subject distance canbe calculated for each calculation block, it is possible to obtain thesubject distance at any arbitrary image position in the color image.Thus, it is also possible to obtain a subject distance map across theentire image. Moreover, the distance from the subject can be obtainedwith a single imaging system, which is unlike in an imaging apparatushaving a plurality of imaging optical systems, where it would benecessary to ensure matching characteristics and positions between theplurality of imaging optical systems. Moreover, when a motion video isshot by using the imaging apparatus of the present embodiment, anaccurate distance from the subject can be measured even if the subjectposition changes with lapse of time.

Moreover, refocusing can be performed for each of the R, G, and Bcomponents, similarly to Embodiment 4. Specifically, at step ST1 shownin FIG. 22, luminance information (e.g. sharpness) is determined foreach of RBG, and as necessary, the color having a low sharpness amongRBG is sharpened. Next, at step ST2, the distance from the subject isdetermined. Furthermore, by using the color image generated at the firstsignal processing section C1, a depth map is generated. Next, at stepST3, a PSF is generated for each subject position, based on the bestfocus position. Herein, one PSF may be generated for the three colors ofRGB. However, in order to take axial chromatic aberration or the likeinto consideration, a PSF may be generated for each of RGB. Next, atstep ST4, a color refocused image at any arbitrary subject position canbe generated.

Embodiment 7

This Embodiment 7 differs from Embodiment 6 in that the areas of dividedregions of the optical element L1 are different, and that the arrayoptical device is changed from lenticular elements to microlenses. Inthe present embodiment, any detailed description directed to similarsubject matter to Embodiments 4 to 6 will be omitted.

FIG. 42 is a front view of the optical element L1 as viewed from thesubject side, the optical element L1 being divided into optical regionsD1 and D2. The optical region D2 is further divided into opticalsubregions d2A, d2B, and d2C. As shown in FIG. 42, the optical region D1and the optical subregions d2A, d2B, and d2C are four divided,upper-lower/right-left parts around the optical axis V as a center ofboundary, in a plane which is perpendicular to the optical axis V. Theoptical regions D1 and D2 have optical characteristics which providemutually different focusing characteristics.

FIG. 43 is a perspective view of the array optical device K. On the faceof the array optical device K closer to the imaging device N, opticalelements M2 are provided in a lattice form. Each optical element M2 hascross sections (cross sections along the vertical direction and alongthe lateral direction) in arc shapes, each optical element M2 protrudingtoward the imaging device N. Thus, the optical elements M2 aremicrolenses, and the array optical device K is a microlens array.

FIG. 44A is an enlarged diagram showing the array optical device K andthe imaging device N, and FIG. 44B is a diagram showing relativepositioning between the array optical device K and pixels on the imagingdevice N. Similarly to Embodiment 4, the array optical device K isdisposed near the focal point of the lens optical system L, being at aposition which is a predetermined distance away from the imaging planeNi. Microlenses Ms are provided on the imaging plane Ni so as to coverthe surface of the pixels P1, P2, P3, and P4.

On the pixels P1, P2, P3, and P4, filters having the same spectraltransmittance characteristics as those in Embodiment 6 are respectivelyprovided.

Moreover, the array optical device K is disposed so that the face onwhich the optical elements M2 are formed is oriented toward the imagingplane Ni. The array optical device K is arranged so that one opticalelement M2 thereof corresponds to four pixels, i.e., two rows by twocolumns of pixels P1 to P4, on the imaging plane Ni.

With such a construction, light beams having passed through the opticalregion D1 and the optical subregions d2A, d2B, and d2C of the opticalelement L1 shown in FIG. 42 mostly reach the pixel P1, the pixel P2, thepixel P3, and the pixel P4 on the imaging plane Ni, respectively.

Similarly to Embodiment 6, the first signal processing section C1generates a color image by using luminance information from theplurality of pixels P1, P2, P3, and P4. Hereinafter, the specific methodof color image generation will be described.

In FIG. 42, the optical region D1 has a non-spherical surface, whereasthe optical subregions d2A, d2B, and d2C all have planar surfaces. Forsimplicity of description, it is assumed that the lens L2 is an ideallens free of aberration.

Due to the aspherical shape of the optical region D1, similarly toEmbodiment 4, in a predetermined range near the focal point of the lensoptical system L, the point spread distribution associated with rayshaving passed through the optical region D1 can be kept substantiallyconstant. In other words, the point spread distribution can be keptsubstantially constant within the predetermined subject distance range.

Since the optical region D2 has a planar surface, no sphericalaberration occurs, similarly to Embodiment 6. When there is no sphericalaberration, the point spread distribution varies with an increase inshift from the focal point. In other words, the point spreaddistribution varies with changing subject distance.

Similarly to Embodiment 6, a graph indication of the relationshipbetween subject distance and sharpness will result in a relationship asshown in FIG. 45. In the graph of FIG. 45, G1 represents sharpness of ina predetermined region of an image generated at the pixel P1 (greencomponent), whereas G2, R, and B respectively represent sharpnesses in apredetermined region of images generated at the pixel P2 (greencomponent), the P3 (red component), and the P4 (blue component).

When generating a color image, similarly to Embodiment 6, the colorimage may be generated by simply interpolating the chromatic informationthat is lost for each pixel position on the basis of the luminanceinformation of the pixels P1, P2, P3, and P4; however, the sharpness ofG1 is smaller than the sharpnesses of G2, R, and B as shown in FIG. 45,and therefore the color image may be generated after enhancing thesharpness of G1, similarly to the method described in FIG. 26.

Through the above image processing, the sharpness of G1 which isindicated by a solid line in FIG. 45 can be enhanced as in G1′ which isindicated by a broken line, thus sharpening the resultant color image.

FIG. 46 is a graph showing the relationship between subject distance andsharpness in the case where the optical surface in the optical region D1is changed from an aspherical shape to a spherical shape in FIG. 45. Inthis case, too, the color image can be sharpened similarly to FIG. 45.

In the present embodiment, as shown in FIG. 46, different colorcomponents have a high sharpness depending on the subject distance.Therefore, respective sharpnesses are detected from the luminance crosssections of G1, G2, R, and B, and the color component with the highestsharpness is selected to sharpen any other color component.

Through the above image processing, the sharpnesses of G1, G2, R, and Bwhich are indicated by solid lines in FIG. 46 can be respectivelysharpened as in G1′, G2′, R′, and B′ which are indicated by brokenlines, thus sharpening the resultant color image.

Next, another image sharpening technique will be described. FIG. 46 is adiagram describing a method of enhancing the sharpnesses of G2, R, and Bbased on G1′, which is a sharpness-enhanced version of G1. Theconstruction of the optical region D1 is the same as that in FIG. 45,and the point spread distribution created by rays having passed throughthe optical region D1 is substantially constant within a predeterminedsubject distance range. Therefore, the point spread distribution whichis created by extracting the pixel P1 (G1 component) is substantiallyconstant within a predetermined subject distance range. So long as thepoint spread distribution is substantially constant in the predeterminedsubject distance range, an image which is formed by extracting from thepixel P1 (G1 component) is restorable based on a predetermined pointspread distribution, regardless of the subject distance.

With the restoration filter described in Embodiment 6, the sharpness ofG1 which is indicated by a solid line in FIG. 47 can be sharpened as inG1′ which is indicated by a dotted line. Furthermore, in a mannersimilar to the method shown in FIG. 26, the G1′ luminance cross sectionmay be subjected to second-order differentiation; and this may besubtracted from G2, R, and B, whereby the sharpnesses of G2, R, and Bare enhanced to result in the sharpened G2′, R′, and B′ which areindicated by a broken line in FIG. 47.

Although the optical element L1 and the lens L2 are separate in thepresent embodiment, another possible construction is where the lens L2has the optical regions D1 and D2, with the optical element L1 beingeliminated. In this case, the stop S may be disposed near the opticalregions D1 and D2 of the lens L2.

Although the present embodiment has illustrated the lens L2 to be anideal lens for simplicity of description as mentioned above, it is notnecessary to employ an ideal lens. For example, although a non-ideallens would have axial chromatic aberration, the axial chromaticaberration may be corrected for by the optical element L1. In thepresent embodiment, FIG. 42 illustrates that the optical regions d2A,d2B, and d2C of the optical element L1 all have planar surfaces;however, they may respectively have different optical surfaces tocorrect for axial chromatic aberration. As described earlier, rayshaving passed through the optical subregions d2A, d2B, and d2C reach thepixel P2, the pixel P3, and the pixel P4 respectively. Since the pixelP2, the pixel P3, and the pixel P4 have filters that mainly allowwavelength components of green, red, and blue to pass through, in thecase where a lens having axial chromatic aberration is adopted for thelens L2, the optical subregions d2A, d2B, and d2C may be allowed to havedifferent optical powers on the respective region surfaces so that thefocusing position in the wavelength band of the filter provided in eachpixel is identical. With such a construction, as compared to the casewhere the optical subregions d2A, d2B, and d2C have an equal opticalpower, the focusing positions of light transmitted through the opticalsubregions d2A, d2B, and d2C can be brought close to one another,whereby the axial chromatic aberration occurring in the lens L2 can becorrected for by the optical element L1. By correcting for the axialchromatic aberration with the optical element L1, the number of lensescomposing the lens L2 can be reduced, thus downsizing the opticalsystem.

Through the above image processing, the sharpness of G1 the sharpnessesof G2, R, and B which are indicated by solid lines in FIG. 47 can besharpened as in G1′ which is indicated by a dotted line and as in G2′,R′, and B′ which are indicated by a broken line, thus sharpening theresultant color image. Through such a sharpening process, the depth offield can be expanded from the sharpening process described in FIG. 45.

The present embodiment compares to Embodiment 6, with the relationshipbetween sharpness G1 and sharpness G2 being merely reversed, and amethod of measuring the distance from a subject can be similarlyimplemented. Moreover, the method of acquiring a refocused image canalso be similarly implemented to Embodiment 6.

Thus, according to the present embodiment, through (e.g. a singleinstance of) imaging using a single imaging system similar to Embodiment6, both a color image and the subject distance can be obtained, and arefocused image can be generated.

Embodiment 8

This Embodiment 8 differs from Embodiment 7 in that color filters areprovided near the stop and no color filters are provided on the imagingplane. In the present embodiment, any detailed description directed tosimilar subject matter to Embodiments 4 to 7 will be omitted.

FIG. 48 is a schematic diagram showing an imaging apparatus A accordingto Embodiment 8. The imaging apparatus A of the present embodimentincludes: a lens optical system L having an optical axis V; an arrayoptical device K disposed near the focal point of the lens opticalsystem L; an imaging device N; a second signal processing section C2; athird signal processing section C3; a first signal processing sectionC1; and a storage section Me.

In Embodiment 8, similarly to FIG. 42, the optical region D1 of theoptical element L1 has a non-spherical surface, whereas the opticalsubregions d2A, d2B, and d2C all have planar surfaces. Each region ofthe optical element L1 has its own spectral transmittancecharacteristics, such that the optical region D1 and the opticalsubregions d2A, d2B, and d2C have characteristics for transmitting lightof G, G, B, and R, respectively, which are converged respectively ontothe pixels P1, P2, P4, and P3. In the present embodiment, sharpenedimages are generated by using images which are obtained from the pixelsP1 (green component), P2 (green component), P3 (red component), and P4(blue component), and subject distance is measured by using images whichare obtained from P1 (green component) and P2 (green component) tocreate a depth map. Generation of PSF data and the refocus method can beimplemented similarly to Embodiments 4 to 7.

Moreover, filters which transmit light of mutually different wavelengthbands and the optical element L1 for providing different focusingcharacteristic may be disposed separately. In this case, the filters andthe optical element L1 may both be provided near the stop S. The orderin which the filters and the optical element L1 are arranged is notlimited. In this case, the optical region D1 and the optical subregionsd2A, d2B, and d2C are regions including both the optical element L1 andthe filters. In this case, each filter may better be set near eachoptical region, and near the stop. One of the filter and the opticalelement L1 may be formed on the optical surface of the lens L2, which isdisposed near the stop S.

Thus, according to the present embodiment, through (e.g. a singleinstance of) imaging using a single imaging system similar to Embodiment7, both a color image and the subject distance can be obtained, and arefocused image can be generated.

Other Embodiments

Although Embodiments 1 to 8 are implementations in which the opticalsurface of any, optical region is disposed on the subject-side face ofthe optical element L1, each optical surface may be disposed on theimage-side face of the optical element L1.

Although the lens L2 is illustrated as being a single lens, the lens L2may be composed of a plurality of groups or a plurality of lenses.

Moreover, the plurality of optical regions may be created on the lens L2being disposed near the stop.

Moreover, filters which transmit light of mutually different wavelengthbands and the optical element L1 for providing different focusingcharacteristic may be disposed separately. In this case, the filters andthe optical element L1 may both be provided near the stop S. The orderin which the filters and the optical element L1 are arranged is notlimited. In this case, the optical region D1 and the optical subregionsd2A, d2B, and d2C are regions including both the optical element L1 andthe filters. One of the filter and the optical element L1 may be formedon the optical surface of the lens L2, which is disposed near the stopS.

Although the optical element L1 is disposed on the subject side of thestop position, it may be disposed on the image side of the stopposition.

Although Embodiments 1 to 8 above illustrate the lens optical system Lto be an image-side telecentric optical system, it may be an image-sidenontelecentric optical system. FIG. 49A is an enlarged diagram showingthe neighborhood of an imaging section. FIG. 49A shows, within the lightpassing through the array optical device K, only a light beam whichpasses through one optical region. As shown in FIG. 49A, when the lensoptical system L is a nontelecentric optical system, light leaking toadjacent pixels is likely to cause crosstalk. However, by allowing thearray optical device to be offset by A from the pixel array as shown inFIG. 49B, crosstalk can be reduced. Since the incident angle will varydepending on the image height, the offset amount A may be set inaccordance with the incident angle of the light beam onto the imagingplane.

In the case where the lens optical system L is an image-side telecentricoptical system, the optical regions D1 and D2 of the optical element L1have two different radii of curvature, thus resulting in differentmagnifications of the images (the first image I1 and the second imageI2) obtained in the respective regions. When the above-discussedsharpness ratio is calculated for each region of the image, there willbe a discrepancy, off the optical axis, in the predetermined regionsthat are relied on; this makes it impossible to correctly determine asharpness ratio. In this case, a correction may be made so that thefirst image I1 and the second image I2 are substantially equal inmagnification, and then a sharpness ratio between predetermined regionsmay be determined. This makes it possible to correctly determine asharpness ratio between predetermined regions.

Embodiments 1 to 8 are directed to imaging apparatuses having the firstsignal processing section C1, the second signal processing section C2,the third signal processing section C3, and the storage section Me(shown in FIG. 18 and so on). However, the imaging apparatus may lackthese signal processing sections and storage section. In that case, a PCor the like which is external to the imaging apparatus may be used toperform the processes that are performed by the first signal processingsection C1, the second signal processing section C2, and the thirdsignal processing section C3. In other words, a system that includes animaging apparatus having the lens optical system L, the array opticaldevice K, and the imaging device N and includes an external signalprocessing apparatus may also be possible. With the imaging apparatusunder this implementation, luminance information for a color imageoutput and subject distance measurement can be obtained through a singleinstance of imaging using a single imaging optical system. Moreover,through processes performed by the external signal processing section byusing that luminance information, both the multicolor image and thesubject distance can be obtained.

According to the distance measurement method of the present invention,it is not always necessary to utilize a correlation between sharpnessand subject distance. For example, a subject distance may be obtained bysubstituting an ascertained sharpness, contrast, or point image diameterinto an equation expressing the relationship between sharpness,contrast, or point image diameter and subject distance.

Moreover, each optical element (microlens) in the microlens array ofthis Embodiment 3 may have a rotation symmetric shape with respect tothe optical axis of each optical element (microlens). This will bediscussed below in comparison with microlenses of a shape which isrotation-asymmetric with respect to the optical axis.

FIG. 50( a 1) is a perspective view showing a microlens array having ashape which is rotation-asymmetric with respect to the optical axis.Such a microlens array is formed by forming quadrangular prisms ofresist on the array and rounding the corner portions of the resistthrough a heat treatment, and performing patterning by using thisresist. The contours of a microlens shown in FIG. 50( a 1) are shown inFIG. 50( a 2). In a microlens having a rotation-asymmetric shape, thereis a difference in radius of curvature between the vertical and lateraldirections (directions parallel to the four sides of the bottom face ofeach microlens) and oblique directions (diagonal direction of the bottomface of the microlens).

FIG. 50( a 3) is a diagram showing ray tracing simulation results in thecase where the microlenses shown in FIGS. 50( a 1) and (a 2) are appliedto the array optical device according to the present invention. AlthoughFIG. 50( a 3) only shows a light beam which passes through only oneoptical region within the light passing through the array optical deviceK, a microlens of a rotation-asymmetric shape will allow light to leakto adjacent pixels, thus causing crosstalk.

FIG. 50( b 1) is a perspective view showing a microlens array having ashape which is rotation symmetric with respect to the optical axis.Microlenses of such a rotation symmetric shape can be formed on a glassplate or the like by a thermal imprinting or UV imprinting manufacturingmethod.

FIG. 50( b 2) shows contours of a microlens having a rotation symmetricshape. In a microlens having a rotation symmetric shape, the radius ofcurvature is identical between the vertical and lateral directions andoblique directions.

FIG. 50( b 3) is a diagram showing ray tracing simulation results in thecase where the microlenses shown in FIGS. 50( b 1) and (b 2) are appliedto the array optical device according to the present invention. AlthoughFIG. 50( b 3) only shows a light beam which passes through only oneoptical region within the light passing through the array optical deviceK, it can be seen that no crosstalk such as that in FIG. 50( a 3) isoccurring. Thus, crosstalk can be reduced by adopting a rotationsymmetric shape for the microlenses, whereby deterioration in theprecision of distance measurement calculation can be suppressed.

In Embodiments 1 to 8, the pixel P1 and the pixel P3 are adjacent toeach other along an oblique direction; however, as in FIG. 51, the pixelP1 and the pixel P3 may be adjacent along the up-down direction.

Regardless of which of the arrangements of FIGS. 4, 21, and so on thepixels P may have, and regardless of which of the constructions of FIG.2, FIG. 14, FIG. 15, and so on the optical element L1 may have, it isalways the same that light having passed through either one of theoptical regions D1 and D2 is incident on the pixel P2 and that lighthaving passed through the remaining one of the optical regions D1 and D2is incident on the pixel P4.

More preferably, only the light having passed through either one of theoptical regions D1 and D2 is incident on the pixel P2 and only the lighthaving passed through the remaining one of the optical regions D1 and D2is incident on the pixel P4. However, it may be possible for a portionof a light beam having passed through the optical region D1 and theoptical subregions d2A, d2B, and d2C to be incident on regions of theimaging plane Ni other than the pixels, an adjacent pixel, or the like.Therefore, in the present specification and the claims, for example,“allowing only the light having passed through the optical region D1 tobe incident on the pixel P2” means that a large part of the lightentering the pixel P2 (e.g., 80% or more) is light from the opticalregion D1, rather than that no light from the optical region D2 isincident on the pixel P2.

The imaging apparatus disclosed herein is useful for imaging apparatusessuch as digital still cameras or digital camcorders. It is alsoapplicable to distance measuring apparatuses for monitoring thesurroundings or monitoring people riding in an automobile, and distancemeasuring apparatuses for inputting three-dimensional information ingames, PCs, mobile terminals, endoscopes, and so on.

While the present invention has been described with respect to preferredembodiments thereof, it will be apparent to those skilled in the artthat the disclosed invention may be modified in numerous ways and mayassume many embodiments other than those specifically described above.Accordingly, it is intended by the appended claims to cover allmodifications of the invention that fall within the true spirit andscope of the invention.

What is claimed is:
 1. An imaging apparatus comprising: a lens opticalsystem having a first region, a second region, and a third region, thefirst region transmitting light of a first wavelength band, the secondregion transmitting light of the first wavelength band and havingoptical characteristics for providing a different focusingcharacteristic from a focusing characteristic associated with raystransmitted through the first region, and the third region transmittinglight of a second wavelength band different from the first wavelengthband; an imaging device on which light having passed through the lensoptical system is incident, the imaging device having a plurality offirst, second, and third pixels; and a microlens array disposed betweenthe lens optical system and the imaging device, the microlens arraycausing light having passed through the first region to enter theplurality of first pixels, light having passed through the second regionto enter the plurality of second pixels, and light having passed throughthe third region to enter the plurality of third pixels.
 2. The imagingapparatus of claim 1, wherein, the lens optical system further has afourth region transmitting light of a third wavelength band differentfrom the first and second wavelength bands; the imaging device furtherincludes a plurality of fourth pixels; and the microlens array causeslight having passed through the fourth region to enter the plurality offourth pixels.
 3. The imaging apparatus of claim 1, wherein the first,second, and third regions are regions divided around an optical axis ofthe lens optical system.
 4. The imaging apparatus of claim 2, wherein,in the lens optical system, a focusing characteristic associated withrays transmitted through the third region and the fourth region isidentical to either a focusing characteristic associated with raystransmitted through the first region or a focusing characteristicassociated with rays transmitted through a second region.
 5. The imagingapparatus of claim 1, wherein rays are incident on the first, second,and third regions through a single instance of imaging.
 6. The imagingapparatus of claim 2, wherein the first and second regions allow rays ofthe green band to pass through, the third region allows rays of the blueband to pass through, and the fourth region allows rays of the red bandto pass through.
 7. The imaging apparatus of claim 1, wherein, when asubject distance is within a predetermined range, a point spreaddistribution created by light entering the first region is substantiallyconstant, and a point spread distribution created by light entering thesecond region varies in accordance with distance from a subject.
 8. Theimaging apparatus of claim 1, wherein a surface of the first region anda surface of the second region have mutually different radii ofcurvature.
 9. The imaging apparatus of claim 1, wherein, the pluralityof first and second pixels respectively generate first and secondluminance information through a single instance of imaging; and theimaging apparatus further comprises a first signal processing sectionfor generating a first image and a second image by using the first andsecond luminance information.
 10. The imaging apparatus of claim 2,further comprising a first signal processing section including asharpness detection section for detecting a sharpness of at least onepixel component, within luminance information of the plurality of firstto fourth pixels, for each predetermined region in an image; whereinbased on a component of a highest sharpness among the respectivesharpnesses, a luminance information component of another pixel issharpened.
 11. The imaging apparatus of claim 10, wherein, by using apreviously stored point spread function, the first signal processingsection performs a restoration process for an image which is formedbased on luminance information of a pixel reached by light entering thefirst region, and generates a restored sharpened image.
 12. The imagingapparatus of claim 11, wherein, the first signal processing section usesa single said point spread function to perform a restoration process forall regions of an image which is formed based on luminance informationof a pixel reached by light entering the first region, and generate arestored sharpened image.
 13. The imaging apparatus of claim 12,wherein, the first signal processing section includes a sharpnessdetection section for detecting a sharpness for each predeterminedregion in the restored sharpened image, and, based on a sharpness ofeach predetermined region in the restored sharpened image, sharpens aluminance information component of another pixel.
 14. The imagingapparatus of claim 9, further comprising a second signal processingsection for calculating a distance from a subject, wherein the secondsignal processing section calculates a distance from the subject byusing the first image and the second image.
 15. The imaging apparatus ofclaim 14, wherein, when the subject distance is within a certain range,a value of a ratio between a sharpness of the first image and asharpness of the second image has a correlation with the distance fromthe subject; and the second signal processing section calculates thedistance from the subject based on the correlation and the ratio betweenthe sharpness of the first image and the sharpness of the second image.16. The imaging apparatus of claim 14, wherein, the first signalprocessing section includes a contrast detection section for detecting acontrast of the first image obtained from the plurality of first pixelsand a contrast of the second image obtained from the plurality of secondpixels and; when the subject distance is within a certain range, a ratiobetween the contrast of the first image and the contrast of the secondimage has a correlation with the subject distance; and the second signalprocessing section calculates the distance from the subject based on thecorrelation, the contrast of the first image and the contrast of thesecond image.
 17. The imaging apparatus of claim 14, wherein the secondsignal processing section calculates the distance from the subject byusing luminance information of an image obtained through addition of thefirst image and the second image and luminance information of the firstimage or the second image.
 18. The imaging apparatus of claim 13,wherein, when the subject distance is within a certain range, a pointspread function derived from an image which is formed from the restoredsharpened image and light entering the second region has a correlationwith the subject distance; and the imaging apparatus further comprisinga second signal processing section for calculating the distance from thesubject based on the correlation and the point spread function.
 19. Theimaging apparatus of claim 2, wherein, the second region, the thirdregion, and the fourth region have mutually different optical powers;and focusing positions of light transmitted through the second region,the third region, and the fourth region are closer to one another thanwhen the second region, the third region, and the fourth region have anequal optical power to one another.
 20. The imaging apparatus of claim1, further comprising a light-shielding member provided at a boundarybetween the first region and the second region.
 21. The imagingapparatus of claim 1, wherein, the lens optical system further includesa stop; and the first region and the second region are disposed near thestop.
 22. The imaging apparatus of claim 14, wherein, the second signalprocessing section calculates a subject distance for each predeterminedregion in an image; and the imaging apparatus further comprises a thirdsignal processing section for generating a refocused image by using thesubject distance for each predetermined region calculated by the secondsignal processing section.
 23. The imaging apparatus of claim 22,wherein the second signal processing section generates a point spreadfunction for each subject distance by using a subject distance for eachpredetermined region.
 24. The imaging apparatus of claim 23, wherein,along the subject distance direction, an intensity change in the pointspread function decreases away from at least one best focus position,the at least one best focus position defining a subject distance atwhich an intensity change in the point spread function takes a localmaximum.
 25. The imaging apparatus of claim 24, wherein the at least onebest focus position is an externally input position or a positiondetermined by the second signal processing section.
 26. The imagingapparatus of claim 23, wherein the third signal processing sectiongenerates the refocused image by using the subject distance for eachpredetermined region and the point spread function.
 27. The imagingapparatus of claim 23, wherein the point spread function is a Gaussianfunction.
 28. The imaging apparatus of claim 26, wherein the thirdsignal processing section generates the refocused image by performing aconvolution calculation for the point spread function using a Fouriertransform for each predetermined region.
 29. The imaging apparatus ofclaim 22, wherein the third signal processing section generates therefocused image by performing a spatial filter process based on thesubject distance for each predetermined region.
 30. The imagingapparatus of claim 24, wherein the at least one best focus positionexists in plurality and discretely.
 31. The imaging apparatus of claim2, further comprising first to fourth filters near the lens opticalsystem, the first to fourth filters being provided respectively in thefirst region, the second region, the third region, and the fourthregion, wherein, the first filter transmits light of the firstwavelength band; the second filter transmits light of the firstwavelength band; the third filter transmits light of the secondwavelength band; and the fourth filter transmits light of the thirdwavelength band.
 32. The imaging apparatus of claim 31, wherein, thelens optical system further comprises a stop; and the first to fourthfilters are disposed near the stop.
 33. An imaging system comprising:the imaging apparatus of claim 2; and a first signal processingapparatus for generating a color image, wherein the first signalprocessing apparatus generates the color image by using luminanceinformation of the plurality of first pixels, the plurality of secondpixels, the plurality of third pixels, and the plurality of fourthpixels obtained through a single instance of imaging.
 34. The imagingsystem of claim 33, further comprising a second signal processingapparatus for calculating a distance from a subject, wherein the secondsignal processing apparatus calculates a distance from the subject byusing the luminance information of the plurality of first pixels and theplurality of second pixels obtained through the single instance ofimaging.
 35. An imaging system comprising an imaging apparatus and asignal processing apparatus, wherein the imaging apparatus includes: alens optical system having a first region and a second region, thesecond region having optical characteristics for providing a differentfocusing characteristic from a focusing characteristic associated withrays having passed through the first region; an imaging device on whichlight having passed through the lens optical system is incident, theimaging device at least having a plurality of first pixels and aplurality of second pixels; and an array optical device disposed betweenthe lens optical system and the imaging device, the array optical devicecausing light having passed through the first region to enter theplurality of first pixels and light having passed through the secondregion to enter the plurality of second pixels, and the signalprocessing apparatus includes: a first signal processing section forcalculating a subject distance for each predetermined region in acaptured image, by using luminance information of a first image obtainedfrom the plurality of first pixels and a second image obtained from theplurality of second pixels; and a second signal processing section forgenerating a refocused image by using the subject distance for eachpredetermined region calculated by the first signal processing section.